TL;DR — Quick Fix
Start with the lowest-risk self-healing — automatic pod restart on OOMKill:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
template:
spec:
restartPolicy: Always
containers:
- name: app
resources:
limits:
memory: "512Mi"
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet:
path: /ready
port: 8080
periodSeconds: 5
failureThreshold: 2
---
Identifying Automatable Incidents
Not every incident should be auto-remediated. Focus on repetitive, well-understood faults.
---
Event-Driven Healing with Argo Events
# argo-events/event-source.yaml
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
name: k8s-pod-events
spec:
resource:
pod-oomkill:
namespace: production
group: ""
version: v1
resource: events
eventTypes:
- ADD
filter:
fields:
- key: reason
operation: "=="
value: "OOMKilled"
---
# argo-events/sensor.yaml
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
name: oom-auto-remediate
spec:
dependencies:
- name: oom-event
eventSourceName: k8s-pod-events
eventName: pod-oomkill
triggers:
- template:
name: restart-deployment
k8s:
operation: patch
source:
resource:
apiVersion: apps/v1
kind: Deployment
parameters:
- src:
dependencyName: oom-event
dataKey: body.involvedObject.name
dest: metadata.name
---
AWS EventBridge + Lambda Pattern
# lambda/auto_remediate.py
import boto3
import json
ecs_client = boto3.client('ecs')
cloudwatch = boto3.client('cloudwatch')
def handler(event, context):
"""Auto-restart unhealthy ECS tasks."""
detail = event.get('detail', {})
cluster = detail.get('clusterArn', '')
stopped_reason = detail.get('stoppedReason', '')
safe_reasons = ['OutOfMemoryError', 'Essential container exited']
if not any(reason in stopped_reason for reason in safe_reasons):
return {'action': 'skipped', 'reason': 'Unknown failure'}
service_name = detail.get('group', '').replace('service:', '')
ecs_client.update_service(
cluster=cluster.split('/')[-1],
service=service_name,
forceNewDeployment=True
)
cloudwatch.put_metric_data(
Namespace='SelfHealing',
MetricData=[{
'MetricName': 'AutoRemediations',
'Value': 1,
'Unit': 'Count',
'Dimensions': [
{'Name': 'Service', 'Value': service_name}
]
}]
)
return {'action': 'remediated', 'service': service_name}
# EventBridge rule (Terraform)
resource "aws_cloudwatch_event_rule" "ecs_task_stopped" {
name = "ecs-task-stopped-auto-heal"
event_pattern = jsonencode({
source = ["aws.ecs"]
detail-type = ["ECS Task State Change"]
detail = {
lastStatus = ["STOPPED"]
desiredStatus = ["STOPPED"]
}
})
}
resource "aws_cloudwatch_event_target" "lambda" {
rule = aws_cloudwatch_event_rule.ecs_task_stopped.name
arn = aws_lambda_function.auto_remediate.arn
}
---
Safety Guardrails
# Prevent remediation loops with rate limiting
apiVersion: v1
kind: ConfigMap
metadata:
name: self-healing-config
data:
max_remediations_per_hour: "3"
cooldown_seconds: "300"
allowed_namespaces: "production,staging"
blocked_services: "database,payment-processor"
require_approval_above: "5" # Human approval after 5 auto-remediations
#!/bin/bash
# safety-check.sh — Called before any auto-remediation
SERVICE=$1
NAMESPACE=$2
# Check if service is in blocked list
BLOCKED=$(kubectl get configmap self-healing-config -o jsonpath='{.data.blocked_services}')
if echo "$BLOCKED" | grep -q "$SERVICE"; then
echo "BLOCKED: $SERVICE requires manual intervention"
exit 1
fi
# Check remediation count in last hour
COUNT=$(kubectl get events -n "$NAMESPACE" \
--field-selector reason=AutoRemediated \
--sort-by='.lastTimestamp' | \
awk -v cutoff="$(date -d '1 hour ago' +%s)" '{print}' | wc -l)
MAX=$(kubectl get configmap self-healing-config -o jsonpath='{.data.max_remediations_per_hour}')
if [[ "$COUNT" -ge "$MAX" ]]; then
echo "RATE LIMITED: $COUNT remediations in last hour (max: $MAX)"
exit 1
fi
echo "APPROVED: Proceeding with remediation for $SERVICE"
exit 0
---
FAQ
Q: How do I prevent auto-remediation loops?
A: Implement three safeguards: (1) cooldown timer between remediations (5+ minutes), (2) max remediations per hour per service, (3) circuit breaker that pages humans after N consecutive auto-remediations.
Q: Should I auto-remediate in production immediately?
A: Start in staging. Run self-healing in "dry-run" mode in production first — log what would have been done. After 2-4 weeks of verified accuracy, enable actual remediation.
Q: What metrics should I track for self-healing?
A: Track: remediations per service per day, time-to-remediation, false positive rate, and incidents that still required human intervention after auto-remediation. Aim for <5% false positive rate.
Q: How does this reduce on-call burden?
A: In practice, teams report 40-60% reduction in pages after implementing self-healing for the top 5 recurring incident types. The remaining pages are genuinely novel problems requiring human judgment.
Q: What about compliance and audit trails?
A: Log every auto-remediation action with timestamp, trigger event, action taken, and result. Store in an immutable audit log. Most compliance frameworks accept automated remediation if the audit trail is complete.
---