Kubernetes OOMKilled (Exit Code 137) — Complete Debugging & Fix Guide

TL;DR — Quick Fix

# 1. Confirm OOMKilled
kubectl describe pod <pod-name> | grep -A3 "Last State"
# Look for: Reason: OOMKilled, Exit Code: 137

# 2. Check current memory usage
kubectl top pod <pod-name>

# 3. Increase memory limit (temporary fix)
kubectl patch deployment <name> -p '{"spec":{"template":{"spec":{"containers":[{"name":"<container>","resources":{"limits":{"memory":"1Gi"}}}]}}}}'

But increasing memory is usually a band-aid. Read on for the real fix.

---

What Is OOMKilled?

When a container tries to allocate memory beyond its configured resources.limits.memory, the Linux kernel's OOM (Out of Memory) killer sends SIGKILL (signal 9) to the process. Kubernetes reports this as Exit Code 137 (128 + 9).

This is not a graceful shutdown. Your application gets zero warning and zero chance to flush buffers or close connections.

Step 1: Confirm the OOMKill

# Get pod status with restart reason
kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
# Output: OOMKilled

# Full details
kubectl describe pod <pod-name>

Look for this in the output:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      Fri, 01 Aug 2026 14:23:01 +0000
  Finished:     Fri, 01 Aug 2026 14:47:33 +0000

Step 2: Understand Your Current Memory Configuration

# Check configured limits
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[0].resources}'

# Check actual usage RIGHT NOW
kubectl top pod <pod-name> --containers

# Check node-level memory pressure
kubectl describe node <node-name> | grep -A5 "Conditions"

Critical distinction:

Step 3: Identify the Root Cause

OOMKilled is a symptom, not a root cause. There are three patterns:

Pattern A: Limit Too Low (Genuine Need)

The application legitimately needs more memory than you've configured.

# Profile actual memory usage over 24 hours
kubectl top pod <pod-name> --containers

# Or use Prometheus query
container_memory_working_set_bytes{pod="<pod-name>"} / 1024 / 1024

Fix:

resources:
  requests:
    memory: "512Mi"   # p50 of actual usage
  limits:
    memory: "1Gi"     # p99 + 30% headroom

Pattern B: Memory Leak (Most Common)

Memory grows continuously until OOM. Restarts temporarily "fix" it.

How to confirm:

# Prometheus query — look for a sawtooth pattern
rate(container_memory_working_set_bytes{pod=~"your-app-.*"}[5m])

If memory grows linearly over time and resets on restart — it's a leak.

Common leak sources by language:

Pattern C: JVM / Runtime Over-Allocation

The runtime allocates more memory than the container limit allows.

Java (most common offender):

env:
  - name: JAVA_OPTS
    value: "-XX:MaxRAMPercentage=75.0 -XX:+UseContainerSupport"

The JVM's default max heap can exceed container limits. -XX:MaxRAMPercentage=75.0 tells the JVM to use at most 75% of the container's memory limit, leaving 25% for non-heap memory (metaspace, thread stacks, native memory).

Go:

env:
  - name: GOMEMLIMIT
    value: "400MiB"  # Set to ~80% of memory limit

Node.js:

env:
  - name: NODE_OPTIONS
    value: "--max-old-space-size=384"  # In MB, set to ~75% of limit

Step 4: Right-Size with VPA (Vertical Pod Autoscaler)

Instead of guessing, let VPA recommend limits based on actual usage:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Off"  # Start with recommendations only
  resourcePolicy:
    containerPolicies:
      - containerName: '*'
        minAllowed:
          memory: "128Mi"
        maxAllowed:
          memory: "2Gi"

Check recommendations:

kubectl describe vpa my-app-vpa
# Look at "Recommendation" section for target, lower bound, upper bound

Step 5: Set Up Alerting Before OOMKill Happens

Don't wait for the kill. Alert when memory usage approaches the limit:

# Prometheus alert rule
groups:
  - name: memory-alerts
    rules:
      - alert: ContainerMemoryNearLimit
        expr: |
          (container_memory_working_set_bytes / container_spec_memory_limit_bytes) > 0.85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Container {{ $labels.container }} in pod {{ $labels.pod }} using >85% of memory limit"

Production Memory Configuration Strategy

# Template for most stateless services
resources:
  requests:
    memory: "256Mi"    # What you normally use (p50)
    cpu: "100m"
  limits:
    memory: "512Mi"    # 2x requests gives burst room
    # cpu: don't set CPU limits (causes throttling)

Rules of thumb:

• Set limits.memory to 1.5x–2x of requests.memory
• Set requests.memory based on p50 actual usage
• Never set requests = limits unless you want Guaranteed QoS (which reduces scheduling flexibility)
• Don't set CPU limits (they cause throttling, not kills)

Common Mistakes

Mistake 1: Setting limits too tight

# Bad — no room for GC spikes, temporary allocations
resources:
  requests:
    memory: "500Mi"
  limits:
    memory: "500Mi"

Mistake 2: Ignoring non-heap memory in JVM apps

# Bad — JVM needs metaspace, thread stacks, native memory
env:
  - name: JAVA_OPTS
    value: "-Xmx450m"  # Container limit is 512Mi — leaves only 62Mi for everything else

Mistake 3: Not accounting for sidecar containers

# Check all containers in the pod, not just the main one
kubectl top pod <pod-name> --containers

---

Frequently Asked Questions

What does exit code 137 mean in Kubernetes?

Exit code 137 means the container process was killed by SIGKILL (signal 9). In Kubernetes, this almost always indicates OOMKilled — the container exceeded its memory limit and the Linux kernel's OOM killer terminated it. The formula is 128 + signal number (128 + 9 = 137).

Should I just increase the memory limit to fix OOMKilled?

Increasing the limit is a valid temporary fix, but investigate the root cause first. If memory grows continuously until OOM (sawtooth pattern on restart), you have a memory leak. Increasing the limit only delays the crash. Profile your application's memory usage to determine whether it genuinely needs more memory or has a leak.

How do I prevent OOMKilled in Java applications?

Set -XX:MaxRAMPercentage=75.0 and ensure -XX:+UseContainerSupport is enabled (default since Java 10). This tells the JVM to use at most 75% of the container's memory limit for heap, leaving 25% for metaspace, thread stacks, and native memory. Never hard-code -Xmx without accounting for the container limit.

What's the difference between memory requests and limits?

Requests tell the Kubernetes scheduler how much memory to reserve on a node when placing the pod. Limits set the maximum memory the container can use before being OOMKilled. Requests affect scheduling; limits affect runtime enforcement. Set requests to your normal usage (p50) and limits to your peak usage plus headroom.

How do I detect memory leaks in Kubernetes pods?

Monitor container_memory_working_set_bytes over time in Prometheus. If memory grows linearly and only resets on pod restart (sawtooth pattern), you have a leak. Use language-specific tools: jmap for Java heap dumps, pprof for Go, --inspect with Chrome DevTools for Node.js, or tracemalloc for Python.

---

Related Resources

• Kubernetes Resource Limits & Requests — Complete guide to CPU and memory configuration
• Pod Troubleshooting Guide — Debug CrashLoopBackOff, Pending, and other pod failures
• Monitoring & Alerting for Production Systems — Set up Prometheus alerts
• Linux Performance Troubleshooting — System-level memory debugging