Kubernetes Pod Troubleshooting: A Senior Engineer's Field Guide

Why Pods Fail — The Real Reasons

Every Kubernetes engineer has been woken up at 3 AM by a PagerDuty alert about crashing pods. The difference between a junior and senior engineer isn't whether they face these issues — it's how fast they resolve them.

This guide gives you a systematic approach to diagnosing and fixing the 5 most common pod failure states in production.

The Diagnostic Framework

Before diving into specific errors, here's the mental model:

# Step 1: What's the pod status?
kubectl get pods -n <namespace>

# Step 2: What happened recently?
kubectl describe pod <pod-name> -n <namespace>

# Step 3: What are the logs saying?
kubectl logs <pod-name> -n <namespace> --previous

# Step 4: What events are firing?
kubectl get events -n <namespace> --sort-by='.lastTimestamp'

Always follow this order. Don't jump to logs before understanding the pod's state.

1. CrashLoopBackOff

This means your container starts, crashes, and Kubernetes keeps restarting it with exponential backoff.

Common causes:

• Application throws an unhandled exception at startup
• Missing environment variable or config file
• Database connection refused (dependency not ready)
• Insufficient memory (OOM before Kubernetes even reports it)

Diagnosis:

# Check the previous container's logs (the one that crashed)
kubectl logs <pod-name> --previous

# Check the exit code
kubectl describe pod <pod-name> | grep -A5 "Last State"

Exit code meanings:

• Exit Code 1 — Application error (check your code)
• Exit Code 137 — OOMKilled (container exceeded memory limit)
• Exit Code 139 — Segfault (memory corruption, bad binary)
• Exit Code 143 — SIGTERM (graceful shutdown failed)

Fix pattern:

# Add an init container to wait for dependencies
initContainers:
  - name: wait-for-db
    image: busybox:1.36
    command: ['sh', '-c', 'until nc -z postgres-svc 5432; do echo waiting for db; sleep 2; done']

2. ImagePullBackOff

Kubernetes can't download your container image.

Common causes:

• Wrong image tag (typo in the tag name)
• Private registry without imagePullSecrets
• Registry rate limiting (Docker Hub free tier: 100 pulls/6 hours)
• Image doesn't exist (deleted or wrong repository)

Diagnosis:

kubectl describe pod <pod-name> | grep -A10 "Events"
# Look for "Failed to pull image" messages

Fix for private registries:

# Create a registry secret
kubectl create secret docker-registry regcred \
  --docker-server=<registry-url> \
  --docker-username=<username> \
  --docker-password=<password> \
  -n <namespace>

# Reference it in your pod spec
spec:
  imagePullSecrets:
    - name: regcred

3. Pending Pods

The pod is created but not scheduled to any node.

Common causes:

• Insufficient CPU or memory on all nodes
• Node affinity/taints preventing scheduling
• PersistentVolumeClaim not bound
• ResourceQuota exhausted in the namespace

Diagnosis:

# Check why the scheduler can't place the pod
kubectl describe pod <pod-name> | grep -A5 "Events"

# Check node resources
kubectl top nodes
kubectl describe nodes | grep -A10 "Allocated resources"

Fix — check if you're over-requesting:

# Before: requesting too much
resources:
  requests:
    memory: "2Gi"
    cpu: "1000m"

# After: right-sized based on actual usage
resources:
  requests:
    memory: "512Mi"
    cpu: "250m"
  limits:
    memory: "1Gi"
    cpu: "500m"

4. OOMKilled (Exit Code 137)

The container exceeded its memory limit and was killed by the Linux OOM killer.

Diagnosis:

kubectl describe pod <pod-name> | grep -i "oom\|memory\|killed"

# Check actual memory usage over time
kubectl top pod <pod-name>

Fix strategies:

• Increase memory limits — but only if the workload genuinely needs it
• Fix memory leaks — the real problem in most cases
• Use JVM flags for Java apps: -XX:MaxRAMPercentage=75.0
• Set Go's GOMEMLIMIT for Go services

resources:
  requests:
    memory: "256Mi"
  limits:
    memory: "512Mi"  # Give 2x headroom over requests

Pro tip: Never set requests equal to limits unless you want Guaranteed QoS. For most workloads, use Burstable:

requests:
  memory: "256Mi"  # What you normally use
limits:
  memory: "512Mi"  # What you can burst to

5. CreateContainerConfigError

The pod spec references a ConfigMap, Secret, or ServiceAccount that doesn't exist.

Diagnosis:

kubectl describe pod <pod-name> | grep -i "error\|config\|secret\|mount"

Fix:

# Check if the secret/configmap exists
kubectl get secret <name> -n <namespace>
kubectl get configmap <name> -n <namespace>

# If using envFrom, verify the source exists
kubectl get pod <pod-name> -o yaml | grep -A5 "envFrom\|secretRef\|configMapRef"

Production Debugging Checklist

When you get a production alert, run through this in order:

kubectl get pods — What's the status?

kubectl describe pod — What events and conditions?

kubectl logs --previous — What did the app say before dying?

kubectl top pod — Is it resource-constrained?

kubectl get events --sort-by='.lastTimestamp' — What's happening cluster-wide?

Keep this checklist bookmarked. In production, speed of diagnosis directly reduces your MTTR (Mean Time To Recovery).

Key Takeaways

• Always check the exit code — it tells you the category of failure
• --previous flag on logs shows you what the dead container said
• Most CrashLoopBackOff issues are missing configs or unready dependencies
• OOMKilled usually means a memory leak, not that you need more RAM
• Pending pods are almost always resource or scheduling constraint issues

Master these patterns and you'll resolve 90% of pod failures in under 5 minutes.

---

Frequently Asked Questions

What does CrashLoopBackOff mean in Kubernetes?

CrashLoopBackOff means the container keeps crashing and Kubernetes is applying an exponential backoff delay before restarting it. Common causes include missing environment variables or config, application errors on startup, failed health checks, and OOM kills. Check logs with kubectl logs <pod> --previous to see what the container printed before crashing.

How do I fix ImagePullBackOff errors?

ImagePullBackOff means Kubernetes cannot pull the container image. Verify the image name and tag are correct, check that the image exists in the registry, ensure image pull secrets are configured for private registries, and confirm network connectivity from nodes to the registry. Use kubectl describe pod to see the exact pull error message.

What causes OOMKilled in Kubernetes pods?

OOMKilled (exit code 137) means the container exceeded its memory limit and was terminated by the kernel. This can indicate a memory leak, insufficient memory limits for the workload, or JVM/runtime default heap sizes exceeding the container limit. Fix by increasing memory limits, fixing leaks, or tuning garbage collector settings to respect container memory boundaries.

Why is my pod stuck in Pending state?

Pending means the pod cannot be scheduled to a node. Common reasons include: insufficient CPU or memory available on any node, PersistentVolumeClaim waiting to be bound, node selectors or affinity rules that no node satisfies, or pod tolerations missing for tainted nodes. Check kubectl describe pod events and kubectl get nodes resource capacity.

How do I debug a pod that won't start?

Use this sequence: kubectl get pod for status, kubectl describe pod for events and conditions, kubectl logs --previous for crash logs, and kubectl get events --sort-by='.lastTimestamp' for cluster context. For deeper debugging, use kubectl exec -it <pod> -- /bin/sh if the pod is running, or create a debug pod with kubectl debug.

---

Related Resources

• kubectl Cheatsheet — 71 kubectl commands searchable by task
• DevOps Interview Academy — 60 DevOps interview questions including Kubernetes
• Production Troubleshooting Scenarios — 15 production troubleshooting scenarios
• kubectl get pods Examples — Practical examples for listing and filtering pods
• Kubernetes Resource Limits — Setting CPU and memory limits to prevent pod issues