Production Troubleshooting Playbooks: 10 Step-by-Step Runbooks for On-Call Engineers

# Production Troubleshooting Playbooks: 10 Step-by-Step Runbooks for On-Call Engineers

Getting paged at 3 AM is stressful enough without having to remember exact diagnostic commands or scramble through documentation. These playbooks give you a clear, repeatable process for the most common production incidents. Each one follows the same structure: verify, diagnose, mitigate, resolve, and close out.

How to use these playbooks:

• Start at Step 1 — don't skip verification even if you think you know the cause
• Follow decision trees exactly — they encode lessons from past incidents
• Document everything you do in the incident channel as you go
• Escalate if you hit the time-to-resolve threshold without progress

---

Playbook 1: High CPU Usage (>90% Sustained)

Symptoms: Alert fires for CPU utilization above 90% for more than 5 minutes. Users report slow response times. Load balancer health checks may start failing.

Severity: P2 (P1 if affecting user-facing services)

Time to Resolve: 15–45 minutes

Steps:

1. Verify the Issue

# Check overall CPU usage
top -bn1 | head -20

# Get per-core breakdown
mpstat -P ALL 1 3

# For Kubernetes workloads
kubectl top nodes
kubectl top pods -n <namespace> --sort-by=cpu

Expected output: One or more cores pinned near 100%, or overall system usage above 90%.

2. Identify Root Cause

# Find the top CPU-consuming processes
ps aux --sort=-%cpu | head -20

# Get detailed thread info for the suspect process
top -H -p <PID>

# Check if it's a recent deployment
kubectl rollout history deployment/<name> -n <namespace>

# Check for runaway cron jobs
ps aux | grep cron
journalctl -u cron --since "1 hour ago"

Decision tree:

• If a single process is consuming >80% CPU → Go to Step 3a (process-level fix)
• If multiple processes are high → Check if there's a traffic spike: netstat -an | grep ESTABLISHED | wc -l
• If it's a Java process → Check for GC storms: jstat -gcutil <PID> 1000 5
• If it's a Node.js process → Possible event loop blocking; check with kill -USR1 <PID> to enable debug

3. Immediate Mitigation

3a — Single runaway process:

# Reduce priority (won't kill it)
renice +10 -p <PID>

# If safe to restart the service
systemctl restart <service-name>

# For Kubernetes — kill the offending pod (it will reschedule)
kubectl delete pod <pod-name> -n <namespace>

3b — Traffic spike causing high CPU:

# Scale horizontally in Kubernetes
kubectl scale deployment/<name> -n <namespace> --replicas=<current+2>

# Enable rate limiting on the load balancer (AWS ALB example)
aws elbv2 modify-rule --rule-arn <arn> --conditions Field=http-request-method

4. Permanent Resolution

• Profile the application under load to find hot paths
• Add CPU resource limits in Kubernetes: resources.limits.cpu
• Implement autoscaling: kubectl autoscale deployment/<name> --min=2 --max=10 --cpu-percent=70
• Review and optimize algorithmic complexity in application code
• Add caching layers for compute-heavy operations

5. Post-Incident Tasks

• [ ] Update the CPU alert threshold if it was too sensitive
• [ ] Add a Grafana dashboard panel for CPU by pod/process
• [ ] Document what caused the spike in the incident timeline
• [ ] Create a ticket for performance profiling if root cause was application code
• [ ] Verify autoscaling policy is appropriate

---

Playbook 2: Memory Exhaustion / OOM Kills

Symptoms: Pods or processes killed by the kernel OOM killer. dmesg shows "Out of memory: Kill process". Kubernetes shows OOMKilled status. Application becomes unresponsive before dying.

Severity: P1 (service is down or degraded)

Time to Resolve: 15–60 minutes

Steps:

1. Verify the Issue

# Check system memory
free -h

# Check for OOM kills in kernel log
dmesg | grep -i "out of memory" | tail -10
journalctl -k | grep -i oom

# Kubernetes OOM events
kubectl get events -n <namespace> --field-selector reason=OOMKilling
kubectl describe pod <pod-name> -n <namespace> | grep -A5 "Last State"

Expected output: OOMKilled exit code (137), or dmesg entries showing killed processes.

2. Identify Root Cause

# Current memory usage by process
ps aux --sort=-%mem | head -20

# Check memory trend over time (if sar is available)
sar -r -s $(date -d '2 hours ago' +%H:%M:%S)

# For a specific process — check for memory leaks
pmap -x <PID> | tail -1

# Kubernetes — check memory limits vs actual usage
kubectl top pods -n <namespace> --sort-by=memory
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].resources}'

Decision tree:

• If memory grows linearly over time → Memory leak. Go to Step 3a.
• If memory spikes suddenly → Traffic burst or large payload. Go to Step 3b.
• If the container limit is too low → Resource misconfiguration. Go to Step 3c.
• If system has no swap and memory is at physical limit → Add swap as emergency or scale up.

3. Immediate Mitigation

3a — Memory leak (restart and investigate later):

# Restart the service to reclaim memory
systemctl restart <service-name>

# Kubernetes — restart pod
kubectl rollout restart deployment/<name> -n <namespace>

# Capture a heap dump before restarting (Java)
jmap -dump:format=b,file=/tmp/heapdump.hprof <PID>

3b — Traffic burst:

# Scale up replicas to distribute memory load
kubectl scale deployment/<name> -n <namespace> --replicas=<current+3>

# Temporarily increase memory limits
kubectl set resources deployment/<name> -n <namespace> --limits=memory=4Gi

3c — Resource misconfiguration:

# Increase memory limits in deployment spec
kubectl edit deployment/<name> -n <namespace>
# Change resources.limits.memory to appropriate value

# Or patch directly
kubectl patch deployment <name> -n <namespace> -p \
  '{"spec":{"template":{"spec":{"containers":[{"name":"<container>","resources":{"limits":{"memory":"4Gi"}}}]}}}}'

4. Permanent Resolution

• Use memory profiling tools (pprof for Go, VisualVM for Java, heapdump for Node.js)
• Set appropriate memory requests AND limits in Kubernetes manifests
• Implement graceful degradation (reject requests when memory is high)
• Add memory-based HPA (Horizontal Pod Autoscaler)
• Review application for unbounded caches, connection pools, or buffer accumulation

5. Post-Incident Tasks

• [ ] Analyze heap dump to find leak source
• [ ] Set up memory usage alerting at 80% of limit (warning) and 90% (critical)
• [ ] Review pod memory limits across all services
• [ ] Add memory usage to service dashboards
• [ ] Create ticket for memory leak investigation with heap dump attached

---

Playbook 3: Disk Space Critical (>95%)

Symptoms: Disk usage alert fires at 95%+. Applications fail to write logs or temp files. Database refuses writes with "No space left on device". Pods stuck in ContainerCreating state.

Severity: P1 (data loss risk if 100% reached)

Time to Resolve: 10–30 minutes

Steps:

1. Verify the Issue

# Check disk usage on all mount points
df -h

# Check inode usage (can be full even with free space)
df -i

# For Kubernetes persistent volumes
kubectl get pv --sort-by='.status.phase'
kubectl exec -it <pod-name> -n <namespace> -- df -h

Expected output: One or more filesystems at 95%+ usage or inodes at 100%.

2. Identify Root Cause

# Find largest directories from root
du -h --max-depth=1 / 2>/dev/null | sort -hr | head -20

# Find largest files modified in last 24h
find / -type f -mtime -1 -size +100M -exec ls -lh {} \; 2>/dev/null

# Check for unlinked but open files (invisible disk usage)
lsof +L1 | head -20

# Check log file sizes
du -sh /var/log/*
ls -lhS /var/log/ | head -10

# Docker-specific — check image/container disk usage
docker system df

Decision tree:

• If /var/log is the culprit → Log rotation issue. Go to Step 3a.
• If /tmp or application data → Large temp files or data growth. Go to Step 3b.
• If Docker overlay → Docker images/containers consuming space. Go to Step 3c.
• If inodes are full (df -i shows 100%) → Too many small files. Go to Step 3d.

3. Immediate Mitigation

3a — Log files consuming space:

# Truncate large log files (preserves file handle)
truncate -s 0 /var/log/<large-file>.log

# Rotate logs immediately
logrotate -f /etc/logrotate.conf

# Find and compress old logs
find /var/log -name "*.log" -mtime +7 -exec gzip {} \;

3b — Large temp or data files:

# Clean temp files older than 2 days
find /tmp -type f -mtime +2 -delete

# Clean package manager cache
apt-get clean    # Debian/Ubuntu
yum clean all    # RHEL/CentOS

# Remove old kernels (Ubuntu)
apt-get autoremove --purge

3c — Docker disk usage:

# Remove unused images, containers, volumes
docker system prune -af --volumes

# Remove dangling images only (safer)
docker image prune -f

# Check and remove stopped containers
docker container prune -f

3d — Inode exhaustion:

# Find directories with the most files
find / -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -20

# Often caused by session files or cache
find /tmp -type f -mtime +1 -delete
find /var/cache -type f -mtime +7 -delete

4. Permanent Resolution

• Implement log rotation with size limits in /etc/logrotate.d/
• Set up disk usage monitoring with alerts at 80% (warning) and 90% (critical)
• Use emptyDir.sizeLimit for Kubernetes pods temp storage
• Implement PersistentVolume auto-expansion in Kubernetes
• Move logs to centralized logging (ELK/CloudWatch) instead of local disk
• Set Docker log driver limits: --log-opt max-size=50m --log-opt max-file=3

5. Post-Incident Tasks

• [ ] Verify log rotation is configured for all services
• [ ] Add disk space alerts for all mount points
• [ ] Review data retention policies
• [ ] Plan disk expansion or volume migration if usage is legitimately growing
• [ ] Audit which services write the most data

---

Playbook 4: Kubernetes Pod CrashLoopBackOff

Symptoms: Pod repeatedly starts and crashes. kubectl get pods shows CrashLoopBackOff status with increasing restart count. Service is unreachable or partially degraded.

Severity: P1 (if all replicas are crashing) / P2 (if some replicas still healthy)

Time to Resolve: 15–60 minutes

Steps:

1. Verify the Issue

# Check pod status
kubectl get pods -n <namespace> -l app=<app-name>

# Look at restart count and status
kubectl get pods -n <namespace> -o wide | grep CrashLoopBackOff

# Check how many replicas are healthy
kubectl get deployment <name> -n <namespace>

Expected output: Pod(s) showing CrashLoopBackOff or high restart count with Error status.

2. Identify Root Cause

# Get pod events (often shows the reason)
kubectl describe pod <pod-name> -n <namespace> | tail -30

# Check current logs
kubectl logs <pod-name> -n <namespace> --tail=100

# Check PREVIOUS container logs (the one that crashed)
kubectl logs <pod-name> -n <namespace> --previous --tail=100

# Check if it's OOM-killed
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'

# Check recent changes
kubectl rollout history deployment/<name> -n <namespace>

Decision tree:

• If exit code is 137 (OOMKilled) → Memory issue. Use Playbook 2.
• If logs show "connection refused" to a dependency → Dependency is down. Check that service first.
• If logs show config/secret errors → Missing or changed ConfigMap/Secret. Go to Step 3a.
• If logs show application startup errors → Bad deployment. Go to Step 3b.
• If no logs at all → Container fails before writing logs. Go to Step 3c.

3. Immediate Mitigation

3a — ConfigMap/Secret issues:

# Verify the ConfigMap exists
kubectl get configmap -n <namespace>
kubectl get secrets -n <namespace>

# Check if values are correct
kubectl get configmap <name> -n <namespace> -o yaml

# If a secret was accidentally deleted, recreate it
kubectl create secret generic <name> -n <namespace> \
  --from-literal=key=value

3b — Bad deployment (rollback):

# Rollback to the last known good revision
kubectl rollout undo deployment/<name> -n <namespace>

# Rollback to a specific revision
kubectl rollout undo deployment/<name> -n <namespace> --to-revision=<N>

# Verify rollback succeeded
kubectl rollout status deployment/<name> -n <namespace>

3c — Container fails immediately:

# Run the container interactively to debug
kubectl run debug-pod --image=<image> -n <namespace> -it --rm -- /bin/sh

# Check if the entrypoint/command is correct
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[0].command}'

# Check if the image exists and is pullable
kubectl get events -n <namespace> | grep -i "pull"

4. Permanent Resolution

• Add proper health checks (liveness and readiness probes) with appropriate thresholds
• Implement graceful shutdown handling in the application
• Use init containers for dependency checking before main container starts
• Add pre-deployment checks in CI/CD to catch config issues
• Set restartPolicy and terminationGracePeriodSeconds appropriately

5. Post-Incident Tasks

• [ ] Root cause the crash — was it code, config, or infrastructure?
• [ ] Add or improve liveness/readiness probes
• [ ] Ensure deployment has proper rollback strategy defined
• [ ] Add pod disruption budget if not present
• [ ] Review CI/CD pipeline for missing validation steps

---

Playbook 5: Database Connection Timeout

Symptoms: Applications log "connection timeout" or "too many connections" errors to the database. API responses return 500/503 errors. Connection pool exhaustion warnings appear. Increased latency on all database-backed endpoints.

Severity: P1 (most applications depend on database)

Time to Resolve: 15–45 minutes

Steps:

1. Verify the Issue

# Test connectivity from the application host
nc -zv <db-host> <db-port> -w 5

# Check if you can connect with the client
mysql -h <host> -u <user> -p -e "SELECT 1;" 2>&1
# Or for PostgreSQL
psql -h <host> -U <user> -d <db> -c "SELECT 1;" 2>&1

# Check from inside Kubernetes pod
kubectl exec -it <app-pod> -n <namespace> -- nc -zv <db-host> <db-port>

# AWS RDS — check instance status
aws rds describe-db-instances --db-instance-identifier <name> \
  --query 'DBInstances[0].DBInstanceStatus'

Expected output: Connection refused, timeout after 5s, or "too many connections" error.

2. Identify Root Cause

# Check active connections on the database
# MySQL
mysql -e "SHOW STATUS LIKE 'Threads_connected';"
mysql -e "SHOW PROCESSLIST;" | wc -l

# PostgreSQL
psql -c "SELECT count(*) FROM pg_stat_activity;"
psql -c "SELECT state, count(*) FROM pg_stat_activity GROUP BY state;"

# Check for long-running queries
# MySQL
mysql -e "SELECT * FROM information_schema.processlist WHERE TIME > 30 ORDER BY TIME DESC;"

# PostgreSQL
psql -c "SELECT pid, now() - pg_stat_activity.query_start AS duration, query
  FROM pg_stat_activity WHERE state != 'idle' ORDER BY duration DESC LIMIT 10;"

# Check database CPU/memory (AWS RDS)
aws cloudwatch get-metric-statistics --namespace AWS/RDS \
  --metric-name CPUUtilization --dimensions Name=DBInstanceIdentifier,Value=<name> \
  --start-time $(date -u -d '30 minutes ago' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) --period 60 --statistics Average

Decision tree:

• If max_connections reached → Connection pool leak or too many clients. Go to Step 3a.
• If database CPU is >90% → Expensive queries. Go to Step 3b.
• If network connectivity fails → Network/security group issue. Go to Step 3c.
• If database is in maintenance/rebooting → Wait or check RDS events.

3. Immediate Mitigation

3a — Connection exhaustion:

# Kill idle connections (PostgreSQL)
psql -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity
  WHERE state = 'idle' AND query_start < now() - interval '10 minutes';"

# Kill idle connections (MySQL)
mysql -e "SELECT CONCAT('KILL ', id, ';') FROM information_schema.processlist
  WHERE Command = 'Sleep' AND TIME > 600;" | mysql

# Restart application pods to reset connection pools
kubectl rollout restart deployment/<app-name> -n <namespace>

3b — Expensive queries consuming resources:

# Kill the long-running query (PostgreSQL)
psql -c "SELECT pg_terminate_backend(<PID>);"

# Kill the long-running query (MySQL)
mysql -e "KILL <process-id>;"

# If RDS — create a read replica to offload reads
aws rds create-db-instance-read-replica \
  --db-instance-identifier <name>-replica \
  --source-db-instance-identifier <name>

3c — Network/security group issue:

# Check security group rules (AWS)
aws ec2 describe-security-groups --group-ids <sg-id> \
  --query 'SecurityGroups[0].IpPermissions'

# Check if the DB subnet is reachable from app subnet
aws ec2 describe-network-interfaces --filters \
  Name=group-id,Values=<sg-id> --query 'NetworkInterfaces[*].PrivateIpAddress'

# Verify DNS resolution of the DB endpoint
nslookup <db-host>
dig <db-host>

4. Permanent Resolution

• Configure connection pooling (PgBouncer for PostgreSQL, ProxySQL for MySQL)
• Set appropriate max_connections and connection pool sizes
• Add connection timeout and retry logic in application code
• Implement query timeouts (statement_timeout in PostgreSQL)
• Set up slow query logging and regular query optimization reviews
• Use read replicas for read-heavy workloads

5. Post-Incident Tasks

• [ ] Review and optimize slow queries identified during the incident
• [ ] Audit connection pool settings across all services
• [ ] Add monitoring for active connections vs max_connections
• [ ] Set up alerts for connection count at 80% of max
• [ ] Document the database's connection capacity and current client count

---

Playbook 6: SSL/TLS Certificate Expiry

Symptoms: Users see "Your connection is not private" or "NET::ERR_CERT_DATE_INVALID" browser warnings. API clients receive SSL handshake failures. Monitoring shows certificate expiry alerts. Webhooks and integrations start failing.

Severity: P1 (site appears "hacked" to users, trust is broken immediately)

Time to Resolve: 10–30 minutes (if automated), 1–4 hours (if manual)

Steps:

1. Verify the Issue

# Check certificate expiry from outside
echo | openssl s_client -servername <domain> -connect <domain>:443 2>/dev/null | \
  openssl x509 -noout -dates

# Check days until expiry
echo | openssl s_client -servername <domain> -connect <domain>:443 2>/dev/null | \
  openssl x509 -noout -enddate

# Check certificate chain
echo | openssl s_client -servername <domain> -connect <domain>:443 -showcerts 2>/dev/null

# Kubernetes — check cert-manager certificates
kubectl get certificates -A
kubectl describe certificate <name> -n <namespace>

Expected output: notAfter date is in the past, or cert-manager shows Ready: False.

2. Identify Root Cause

# Check if cert-manager is running
kubectl get pods -n cert-manager
kubectl logs -n cert-manager deployment/cert-manager --tail=50

# Check certificate renewal events
kubectl get events -n <namespace> --field-selector involvedObject.kind=Certificate

# Check if Let's Encrypt rate limits were hit
kubectl describe order -n <namespace> | grep -i "error\|failed"

# Check if DNS challenge is working (for wildcard certs)
dig TXT _acme-challenge.<domain>

# AWS ACM — check certificate status
aws acm describe-certificate --certificate-arn <arn> \
  --query 'Certificate.{Status:Status,NotAfter:NotAfter}'

Decision tree:

• If cert-manager renewal failed → Fix the issuer config. Go to Step 3a.
• If Let's Encrypt rate-limited → Use a different ACME provider temporarily. Go to Step 3b.
• If AWS ACM validation failed → DNS or email validation issue. Go to Step 3c.
• If manual certificate (no automation) → Manually renew. Go to Step 3d.

3. Immediate Mitigation

3a — cert-manager renewal fix:

# Delete and recreate the certificate to trigger renewal
kubectl delete certificate <name> -n <namespace>
kubectl apply -f certificate.yaml

# Force renewal by deleting the secret
kubectl delete secret <tls-secret-name> -n <namespace>

# Check the new certificate
kubectl get certificate <name> -n <namespace> -w

3b — Alternative ACME provider:

# Switch to ZeroSSL or BuyPass as temporary issuer
# Update the ClusterIssuer to use alternative ACME server
kubectl edit clusterissuer letsencrypt-prod
# Change server to: https://acme.zerossl.com/v2/DV90

3c — AWS ACM validation:

# Check pending validation
aws acm describe-certificate --certificate-arn <arn> \
  --query 'Certificate.DomainValidationOptions'

# Add the CNAME record for DNS validation
aws route53 change-resource-record-sets --hosted-zone-id <zone-id> \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "<validation-domain>",
        "Type": "CNAME",
        "TTL": 300,
        "ResourceRecords": [{"Value": "<validation-value>"}]
      }
    }]
  }'

3d — Manual certificate renewal:

# Generate CSR and get new certificate from your CA
openssl req -new -key /etc/ssl/private/<domain>.key \
  -out /tmp/<domain>.csr -subj "/CN=<domain>"

# After receiving the new cert, install it
cp /tmp/new-cert.pem /etc/ssl/certs/<domain>.pem
nginx -t && systemctl reload nginx

# Update Kubernetes secret
kubectl create secret tls <name> -n <namespace> \
  --cert=new-cert.pem --key=private.key --dry-run=client -o yaml | kubectl apply -f -

4. Permanent Resolution

• Implement cert-manager with automatic renewal (renew 30 days before expiry)
• Set up certificate expiry monitoring (alert at 30, 14, and 7 days before expiry)
• Use AWS ACM for ALB/CloudFront (auto-renewed by AWS)
• Document all certificates, their locations, and renewal processes
• Automate certificate deployment in CI/CD pipeline

5. Post-Incident Tasks

• [ ] Audit all certificates across the infrastructure for upcoming expiries
• [ ] Set up Prometheus alerting: certmanager_certificate_expiration_timestamp_seconds
• [ ] Ensure cert-manager is configured with proper DNS solvers
• [ ] Add certificate expiry to the team's operational dashboard
• [ ] Review why automated renewal failed and fix the root cause

---

Playbook 7: DNS Resolution Failure

Symptoms: Applications log "Name or service not known" or "NXDOMAIN" errors. External API calls fail. Internal service-to-service communication breaks. Users get "This site can't be reached" in browsers.

Severity: P1 (cascading failures across all services relying on DNS)

Time to Resolve: 10–45 minutes

Steps:

1. Verify the Issue

# Test DNS resolution from the host
nslookup <domain>
dig <domain> +short
host <domain>

# Test from inside a Kubernetes pod
kubectl run dns-test --image=busybox:1.36 --rm -it --restart=Never -- nslookup <service-name>
kubectl run dns-test --image=busybox:1.36 --rm -it --restart=Never -- nslookup <external-domain>

# Check if it's specific to one domain or all DNS
dig google.com +short
dig <internal-service>.svc.cluster.local +short

# Check /etc/resolv.conf
cat /etc/resolv.conf

Expected output: "NXDOMAIN", "SERVFAIL", or timeout when querying the domain.

2. Identify Root Cause

# Check CoreDNS pods (Kubernetes)
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

# Check CoreDNS metrics for errors
kubectl exec -n kube-system <coredns-pod> -- cat /etc/coredns/Corefile

# Check if it's an upstream DNS issue
dig <domain> @8.8.8.8
dig <domain> @1.1.1.1

# Check Route53 (AWS) for the hosted zone
aws route53 list-resource-record-sets --hosted-zone-id <zone-id> \
  --query "ResourceRecordSets[?Name=='<domain>.']"

# Check for DNS propagation issues
dig <domain> +trace

Decision tree:

• If CoreDNS pods are crashed/pending → CoreDNS issue. Go to Step 3a.
• If external DNS works but internal doesn't → Kubernetes DNS config issue. Go to Step 3b.
• If the DNS record doesn't exist → Record was deleted or never created. Go to Step 3c.
• If upstream resolvers timeout → ISP/provider DNS outage. Go to Step 3d.

3. Immediate Mitigation

3a — CoreDNS pods down:

# Restart CoreDNS
kubectl rollout restart deployment/coredns -n kube-system

# Check if CoreDNS has enough resources
kubectl describe deployment coredns -n kube-system | grep -A5 Resources

# Scale CoreDNS if under heavy load
kubectl scale deployment/coredns -n kube-system --replicas=4

3b — Kubernetes DNS configuration:

# Verify the kube-dns service exists
kubectl get svc kube-dns -n kube-system

# Check if pods have correct DNS policy
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.dnsPolicy}'

# Verify the ClusterIP of kube-dns matches resolv.conf in pods
kubectl exec -it <pod> -n <namespace> -- cat /etc/resolv.conf

3c — Missing DNS record:

# Create the missing record (AWS Route53)
aws route53 change-resource-record-sets --hosted-zone-id <zone-id> \
  --change-batch '{
    "Changes": [{
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "<domain>",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [{"Value": "<ip-address>"}]
      }
    }]
  }'

# For Kubernetes services — verify the service exists
kubectl get svc -n <namespace> | grep <service-name>

3d — Upstream DNS outage:

# Switch to alternative DNS servers
# Edit CoreDNS Corefile to use different forwarders
kubectl edit configmap coredns -n kube-system
# Change forward to: forward . 8.8.8.8 1.1.1.1

# For hosts — temporarily change /etc/resolv.conf
echo "nameserver 8.8.8.8" > /etc/resolv.conf
echo "nameserver 1.1.1.1" >> /etc/resolv.conf

4. Permanent Resolution

• Implement DNS caching at the node level (NodeLocal DNSCache)
• Use headless services for internal communication where possible
• Set up DNS monitoring (query success rate, latency)
• Configure multiple upstream DNS forwarders in CoreDNS
• Implement DNS record management through IaC (Terraform/ExternalDNS)
• Add TTL-aware retries in application code for DNS failures

5. Post-Incident Tasks

• [ ] Verify all DNS records are managed through IaC
• [ ] Set up synthetic monitoring for critical DNS records
• [ ] Review CoreDNS resource allocation and scaling
• [ ] Add DNS resolution checks to readiness probes
• [ ] Document all critical DNS records and their dependencies

---

Playbook 8: Application Returning 5xx Errors

Symptoms: Monitoring shows spike in HTTP 500, 502, 503, or 504 errors. Users report "Internal Server Error" pages. Load balancer health checks failing. Error rate exceeds SLO threshold.

Severity: P1 (direct user impact, SLO violation)

Time to Resolve: 15–60 minutes

Steps:

1. Verify the Issue

# Test the endpoint directly
curl -s -o /dev/null -w "%{http_code}" https://<domain>/health
curl -v https://<domain>/api/<endpoint> 2>&1

# Check error rates in the load balancer (AWS ALB)
aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB \
  --metric-name HTTPCode_Target_5XX_Count \
  --dimensions Name=LoadBalancer,Value=<lb-arn-suffix> \
  --start-time $(date -u -d '15 minutes ago' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) --period 60 --statistics Sum

# Check from inside the cluster
kubectl exec -it <pod> -n <namespace> -- curl -s localhost:<port>/health

# Check ingress/service status
kubectl get ingress -n <namespace>
kubectl describe svc <service-name> -n <namespace>

Expected output: HTTP 5xx status codes returned from the service or load balancer.

2. Identify Root Cause

# Check application logs for errors
kubectl logs -n <namespace> -l app=<name> --tail=200 | grep -i "error\|exception\|panic"

# Check if pods are healthy
kubectl get pods -n <namespace> -l app=<name>
kubectl describe pods -n <namespace> -l app=<name> | grep -A3 "Conditions"

# Differentiate 502 vs 503 vs 500
# 502 = upstream unreachable (pod crashed or not ready)
# 503 = service unavailable (no healthy backends)
# 500 = application error (bug in code)

# Check the target group health (AWS)
aws elbv2 describe-target-health --target-group-arn <tg-arn>

# Check recent deployments
kubectl rollout history deployment/<name> -n <namespace>
helm history <release-name> -n <namespace>

Decision tree:

• If 502 (Bad Gateway) → Pods are crashing or not responding. Check Playbook 4.
• If 503 (Service Unavailable) → No healthy backends. Go to Step 3a.
• If 500 (Internal Server Error) → Application bug. Go to Step 3b.
• If 504 (Gateway Timeout) → Downstream dependency slow. Go to Step 3c.

3. Immediate Mitigation

3a — No healthy backends (503):

# Check if readiness probe is failing
kubectl describe pod <pod-name> -n <namespace> | grep -A5 "Readiness"

# Check if the pod can respond on the health endpoint
kubectl exec -it <pod> -n <namespace> -- curl -s localhost:<port>/health

# Scale up if some pods are healthy but overloaded
kubectl scale deployment/<name> -n <namespace> --replicas=<current+3>

# If no pods are starting — check events
kubectl get events -n <namespace> --sort-by='.lastTimestamp' | tail -20

3b — Application error (500):

# Check if a recent deployment caused it
kubectl rollout undo deployment/<name> -n <namespace>

# If it's a specific endpoint, check downstream dependencies
kubectl exec -it <pod> -n <namespace> -- curl -s <dependency-url>/health

# Enable debug logging temporarily
kubectl set env deployment/<name> -n <namespace> LOG_LEVEL=debug

3c — Gateway timeout (504):

# Identify which downstream service is slow
kubectl exec -it <pod> -n <namespace> -- curl -w "\n%{time_total}\n" -s <downstream-url>

# Increase timeout temporarily on the ingress
kubectl annotate ingress <name> -n <namespace> \
  nginx.ingress.kubernetes.io/proxy-read-timeout="120" --overwrite

# Check if the downstream database is slow (refer to Playbook 5)

4. Permanent Resolution

• Implement circuit breakers for downstream dependencies
• Add proper retry logic with exponential backoff
• Set appropriate timeouts at every layer (ingress, service mesh, application)
• Implement graceful degradation (serve cached responses when backends fail)
• Add canary deployments to catch errors before full rollout
• Set up error budget tracking and alerting

5. Post-Incident Tasks

• [ ] Identify the exact error in application logs and create a bug ticket
• [ ] Review and fix the failing health check if it was too aggressive
• [ ] Add error rate monitoring per endpoint (not just aggregate)
• [ ] Set up SLO burn-rate alerts for early detection
• [ ] Conduct postmortem if SLO was breached

---

Playbook 9: Deployment Rollback Required

Symptoms: New deployment causes errors, performance degradation, or feature regressions. Error rates spike immediately after deploy. Canary analysis fails. Users report broken functionality that worked before the deployment.

Severity: P1 (if causing errors) / P2 (if performance only)

Time to Resolve: 5–15 minutes (rollback itself), 1–4 hours (fix forward)

Steps:

1. Verify the Issue

# Confirm the recent deployment
kubectl rollout history deployment/<name> -n <namespace>
helm history <release-name> -n <namespace>

# Check when the errors started vs when deploy happened
# Compare deployment timestamp with error spike in monitoring

# Verify current vs previous revision
kubectl rollout status deployment/<name> -n <namespace>
kubectl get replicasets -n <namespace> -l app=<name> --sort-by='.metadata.creationTimestamp'

# Check if only the new pods are failing
kubectl get pods -n <namespace> -l app=<name> -o wide

Expected output: Error spike correlates exactly with the deployment timestamp.

2. Identify Root Cause

# Compare the current and previous deployment specs
kubectl rollout history deployment/<name> -n <namespace> --revision=<current>
kubectl rollout history deployment/<name> -n <namespace> --revision=<previous>

# Check what changed in the image
# Pull the diff from your CI/CD system or git
git log --oneline <previous-tag>..<current-tag>

# Check if it's a config change, not code
kubectl diff -f <manifest-file>

# Check for environment-specific issues
kubectl get configmap -n <namespace> -o yaml
kubectl get secrets -n <namespace>

Decision tree:

• If errors are widespread and impacting all users → Immediate rollback. Go to Step 3a.
• If only a subset of requests are failing → Possible feature flag or canary issue. Go to Step 3b.
• If performance degraded but no errors → Consider fix-forward if quick. Go to Step 3c.
• If database migration was part of the deploy → Rollback is risky. Go to Step 3d.

3. Immediate Mitigation

3a — Full rollback (Kubernetes):

# Rollback deployment to previous version
kubectl rollout undo deployment/<name> -n <namespace>

# Verify the rollback
kubectl rollout status deployment/<name> -n <namespace>
kubectl get pods -n <namespace> -l app=<name>

# Helm rollback
helm rollback <release-name> <previous-revision> -n <namespace>

# ArgoCD rollback
argocd app rollback <app-name> --revision <previous-revision>

3b — Canary/partial rollback:

# If using Argo Rollouts — abort the canary
kubectl argo rollouts abort <name> -n <namespace>

# If using Istio — shift traffic back to stable
kubectl patch virtualservice <name> -n <namespace> --type merge -p \
  '{"spec":{"http":[{"route":[{"destination":{"host":"<svc>","subset":"stable"},"weight":100}]}]}}'

# Disable feature flag if the issue is behind a flag
# (Use your feature flag service API)

3c — Fix-forward (quick patch):

# Only if the fix is trivial and tested
# Build and push the fix
docker build -t <image>:<hotfix-tag> .
docker push <image>:<hotfix-tag>

# Deploy the hotfix
kubectl set image deployment/<name> -n <namespace> <container>=<image>:<hotfix-tag>

# Watch the rollout
kubectl rollout status deployment/<name> -n <namespace> --timeout=300s

3d — Database migration involved:

# DO NOT rollback if migration is not backward-compatible
# Instead, fix the application code to work with both old and new schema

# If migration is backward-compatible, proceed with app rollback
kubectl rollout undo deployment/<name> -n <namespace>

# If migration must be reverted (DANGEROUS — test first)
# Run the down migration against a copy first
kubectl exec -it <migration-pod> -n <namespace> -- ./migrate down 1

4. Permanent Resolution

• Implement canary deployments with automatic rollback on error threshold
• Add deployment gates (require health check pass before promoting)
• Use progressive delivery tools (Argo Rollouts, Flagger)
• Ensure database migrations are always backward-compatible
• Add pre-deployment smoke tests in the pipeline
• Implement feature flags for risky changes

5. Post-Incident Tasks

• [ ] Identify what test coverage missed the issue
• [ ] Add the failure scenario as a test case
• [ ] Review deployment strategy (rolling vs canary vs blue-green)
• [ ] Set up automatic rollback triggers in CI/CD
• [ ] Update deployment runbook with lessons learned

---

Playbook 10: Network Connectivity Loss Between Services

Symptoms: Service A cannot reach Service B. Intermittent connection timeouts or "connection refused" between internal services. gRPC/HTTP calls between microservices fail. Network policies may be blocking traffic. Service mesh reports upstream connection failures.

Severity: P1 (cascading failure across dependent services)

Time to Resolve: 15–60 minutes

Steps:

1. Verify the Issue

# Test connectivity from source pod to destination
kubectl exec -it <source-pod> -n <namespace> -- nc -zv <dest-service> <port> -w 5
kubectl exec -it <source-pod> -n <namespace> -- curl -s -o /dev/null -w "%{http_code}" http://<dest-service>:<port>/health

# Check if the destination service is reachable via ClusterIP
kubectl get svc <dest-service> -n <dest-namespace>
kubectl get endpoints <dest-service> -n <dest-namespace>

# Verify DNS resolution between namespaces
kubectl exec -it <source-pod> -n <namespace> -- nslookup <dest-service>.<dest-namespace>.svc.cluster.local

# Check from multiple source pods to rule out node-specific issues
kubectl get pods -n <namespace> -o wide | head -5

Expected output: Connection timeout, "connection refused", or empty endpoint list.

2. Identify Root Cause

# Check if the destination pods are running
kubectl get pods -n <dest-namespace> -l app=<dest-app>
kubectl get endpoints <dest-service> -n <dest-namespace> -o yaml

# Check NetworkPolicies that might be blocking traffic
kubectl get networkpolicies -n <dest-namespace>
kubectl describe networkpolicy -n <dest-namespace>

# Check if it's a service mesh issue (Istio)
istioctl analyze -n <namespace>
kubectl get destinationrule -n <namespace>
kubectl get virtualservice -n <namespace>

# Check for node-level network issues
kubectl get nodes -o wide
# From the node: check iptables rules
iptables -t nat -L KUBE-SERVICES | grep <service-name>

# Check for AWS Security Group or NaCL issues
aws ec2 describe-security-groups --group-ids <sg-id>
aws ec2 describe-network-acls --network-acl-ids <nacl-id>

# Check CNI plugin health (Calico/Cilium)
kubectl get pods -n kube-system -l k8s-app=calico-node
kubectl logs -n kube-system -l k8s-app=calico-node --tail=30

Decision tree:

• If endpoints list is empty → Service has no healthy backends. Fix the pods first (Playbook 4).
• If NetworkPolicy is blocking → Policy misconfiguration. Go to Step 3a.
• If connectivity works from some nodes but not others → Node-level networking issue. Go to Step 3b.
• If service mesh (Istio/Linkerd) sidecar is the problem → Mesh config issue. Go to Step 3c.
• If AWS security groups changed → Infrastructure networking. Go to Step 3d.

3. Immediate Mitigation

3a — NetworkPolicy blocking traffic:

# Temporarily allow all ingress to the destination (emergency only)
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-ingress-emergency
  namespace: <dest-namespace>
spec:
  podSelector:
    matchLabels:
      app: <dest-app>
  ingress:
  - {}
  policyTypes:
  - Ingress
EOF

# Or fix the specific policy to allow the source namespace
kubectl edit networkpolicy <policy-name> -n <dest-namespace>
# Add the source namespace/pod labels to the ingress rules

3b — Node-level networking:

# Cordon the problematic node
kubectl cordon <node-name>

# Drain pods off the node (moves them to healthy nodes)
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# Restart the CNI plugin on the affected node
kubectl delete pod -n kube-system -l k8s-app=calico-node --field-selector spec.nodeName=<node-name>

# Check if the node's network interface is healthy
ssh <node> "ip link show && ip route show"

3c — Service mesh configuration:

# Check sidecar injection
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].name}'

# Restart the sidecar proxy
kubectl delete pod <pod-name> -n <namespace>

# Check Istio proxy status
istioctl proxy-status
istioctl proxy-config cluster <pod-name>.<namespace>

# If mTLS is the issue — check PeerAuthentication
kubectl get peerauthentication -A

3d — AWS Security Group / infrastructure:

# Add the missing ingress rule
aws ec2 authorize-security-group-ingress --group-id <sg-id> \
  --protocol tcp --port <port> --source-group <source-sg-id>

# Check VPC peering or Transit Gateway routes
aws ec2 describe-vpc-peering-connections --filters Name=status-code,Values=active
aws ec2 describe-route-tables --route-table-ids <rtb-id>

# If using PrivateLink — check the endpoint
aws ec2 describe-vpc-endpoints --vpc-endpoint-ids <vpce-id>

4. Permanent Resolution

• Implement NetworkPolicies with explicit allow rules (deny by default)
• Use service mesh observability to detect connectivity issues early
• Automate security group management through Terraform
• Implement health checks that verify downstream connectivity
• Set up network topology monitoring (packet loss, latency between nodes)
• Document all inter-service communication paths and required ports

5. Post-Incident Tasks

• [ ] Map all service-to-service communication dependencies
• [ ] Review and document all NetworkPolicies
• [ ] Add network connectivity checks to synthetic monitoring
• [ ] Verify CNI plugin is properly resourced and updated
• [ ] Create a network diagram showing allowed communication paths
• [ ] Set up alerts for endpoint count drops (service has no backends)

---

General Incident Response Framework

Before diving into any specific playbook, follow this framework:

The First 5 Minutes

Acknowledge the alert — Let the team know you're looking at it

Assess blast radius — How many users/services are affected?

Check for correlation — Did anything change recently? (deployments, config changes, traffic spikes)

Open an incident channel — Keep all communication in one place

Start the clock — Note when you were paged and when you started working

Escalation Guidelines

Communication Template

Post updates every 15 minutes during an active incident:

Status: Investigating / Identified / Mitigating / Resolved
Impact: [Who is affected and how]
Current action: [What you're doing right now]
Next step: [What you'll try if current action doesn't work]
ETA: [Best guess, or "unknown"]

Post-Incident Checklist (All Playbooks)

After every incident, regardless of severity:

• [ ] Write a timeline of events (when detected, when mitigated, when resolved)
• [ ] Identify contributing factors (not "root cause" — incidents are usually multi-causal)
• [ ] List action items with owners and due dates
• [ ] Update monitoring/alerting if the issue wasn't caught quickly enough
• [ ] Update this runbook if the playbook was missing steps or had incorrect commands
• [ ] Share learnings with the broader team (blameless postmortem)

---

Quick Reference: Emergency Commands

When you need to act fast and figure out the details later:

# Kubernetes — nuclear options (use with caution)
kubectl rollout undo deployment/<name> -n <namespace>          # Rollback
kubectl scale deployment/<name> -n <namespace> --replicas=0    # Stop all pods
kubectl delete pod --all -n <namespace> -l app=<name>          # Kill all pods (they reschedule)

# Linux — find what's consuming resources NOW
top -bn1 | head -5                    # CPU summary
free -h                               # Memory summary
df -h                                 # Disk summary
ss -tlnp                              # What's listening on ports
netstat -an | grep ESTABLISHED | wc -l # Connection count

# AWS — quick checks
aws sts get-caller-identity                                    # Verify your credentials work
aws rds describe-db-instances --query 'DBInstances[*].{ID:DBInstanceIdentifier,Status:DBInstanceStatus}'
aws ecs describe-services --cluster <cluster> --services <svc> --query 'services[0].{desired:desiredCount,running:runningCount}'

---

Conclusion

These playbooks are living documents. Every incident you handle should make them better. If you find a step that's wrong, a command that doesn't work, or a decision tree that led you down the wrong path — fix it immediately after the incident is resolved.

The goal isn't to eliminate all incidents (that's impossible). The goal is to make each incident shorter, less stressful, and a learning opportunity for the team.

Remember: At 3 AM, the playbook is your friend. Trust the process, follow the steps, escalate early if you're stuck, and document everything.

---

Frequently Asked Questions

What should a production troubleshooting playbook include?

A good playbook includes: symptom description, severity classification, diagnostic commands to run in order, likely root causes with resolution steps, escalation criteria with contact information, and a verification section to confirm the issue is resolved. Include copy-paste ready commands and decision trees for common branch points. Keep playbooks under 2 pages for quick use during incidents.

How do I create effective runbooks for my team?

Write runbooks during or immediately after incidents when procedures are fresh. Use numbered steps with exact commands (no ambiguity), include expected output so operators can verify they're on track, and note what NOT to do. Test runbooks by having someone unfamiliar with the system follow them. Review and update quarterly or after each incident reveals gaps.

What is the difference between a runbook and a playbook?

A runbook is a step-by-step procedure for a specific operational task (restart a service, rotate credentials, scale up). A playbook is broader — it covers an entire incident scenario from detection through diagnosis to resolution, often including multiple runbooks. Think of playbooks as the strategy and runbooks as tactical procedures within that strategy.

How do I reduce Mean Time To Recovery (MTTR)?

Reduce detection time with better monitoring and alerting. Reduce diagnosis time with structured troubleshooting playbooks, centralized logging, and distributed tracing. Reduce resolution time with automated rollback capabilities, pre-written runbooks, and self-healing systems. Practice incident response through regular game days so the team executes efficiently under pressure.