Kubernetes OOMKilled — Setting CPU & Memory Limits That Don't Kill Your Pods

The Problem

Your pod gets OOMKilled at 3 AM. Or a single pod without limits consumes all node memory, taking down every other pod. Resource requests and limits protect the cluster from noisy neighbors and ensure proper scheduling.

Requests vs Limits

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  template:
    spec:
      containers:
        - name: api
          image: myapp:latest
          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              cpu: "1000m"
              memory: "512Mi"

Units

• CPU: 1 = 1 vCPU, 500m = 0.5 CPU, 100m = 0.1 CPU
• Memory: Mi (mebibytes), Gi (gibibytes)

CPU Throttling

When a container exceeds its CPU limit, it is throttled — stays running but becomes slower.

kubectl exec deploy/api-server -- cat /sys/fs/cgroup/cpu/cpu.stat
kubectl top pod api-server-abc123

Signs: increased latency, timeouts, intermittent health check failures.

Key insight: CPU limits cause throttling even when the node has idle CPU. Many teams remove CPU limits to avoid unnecessary throttling.

OOMKill

When a container exceeds its memory limit, the kernel kills it immediately.

kubectl describe pod api-server-abc123 | grep -A5 "Last State"
kubectl get events --field-selector reason=OOMKilling -A

QoS Classes

Guaranteed (requests = limits)

resources:
  requests:
    cpu: "500m"
    memory: "256Mi"
  limits:
    cpu: "500m"
    memory: "256Mi"

Last to be evicted under pressure.

Burstable (requests < limits)

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    cpu: "1000m"
    memory: "512Mi"

BestEffort (no resources set)

First to be evicted. Never use in production.

kubectl get pod my-pod -o jsonpath='{.status.qosClass}'

Right-Sizing with Metrics

kubectl top pods -n production
kubectl top pods --sort-by=memory -n production
kubectl top nodes

VPA recommendations

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-server-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  updatePolicy:
    updateMode: "Off"

Right-sizing workflow

Start with generous limits

Run under production load for 24-48 hours

Check actual usage: kubectl top pods --containers

Set requests to p95 of actual usage

Set limits to 2x requests for bursty workloads

Monitor for throttling and OOMKills

LimitRanges

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: production
spec:
  limits:
    - default:
        cpu: "500m"
        memory: "256Mi"
      defaultRequest:
        cpu: "100m"
        memory: "128Mi"
      max:
        cpu: "4"
        memory: "4Gi"
      min:
        cpu: "50m"
        memory: "64Mi"
      type: Container

ResourceQuotas

apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-quota
  namespace: production
spec:
  hard:
    requests.cpu: "50"
    requests.memory: "100Gi"
    limits.cpu: "100"
    limits.memory: "200Gi"
    pods: "200"

kubectl describe quota production-quota -n production

Workload-Specific Configurations

Web API

resources:
  requests: { cpu: "250m", memory: "256Mi" }
  limits: { cpu: "1000m", memory: "512Mi" }

JVM application

resources:
  requests: { cpu: "500m", memory: "1Gi" }
  limits: { cpu: "2000m", memory: "1536Mi" }
env:
  - name: JAVA_OPTS
    value: "-Xmx1152m -Xms1152m"  # 75% of memory limit

Background worker

resources:
  requests: { cpu: "1000m", memory: "512Mi" }
  limits: { cpu: "2000m", memory: "1Gi" }

Sidecar

resources:
  requests: { cpu: "50m", memory: "64Mi" }
  limits: { cpu: "200m", memory: "128Mi" }

HPA with Resources

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-server-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

HPA uses requests as the 100% baseline. 250m request with 70% target scales at 175m average usage.

Common Mistakes

No resource requests — Scheduler cannot make informed placement decisions. Pods land on overloaded nodes.

Memory limit too close to actual usage — Applications spike. Set limits with headroom above peak.

CPU limits causing unnecessary throttling — Consider removing CPU limits (keep requests) to allow bursting.

JVM heap not aligned with container memory — If limit is 512Mi but heap is unconfigured, JVM gets OOMKilled.

Not accounting for init containers — Init containers need resources for scheduling even though they run before main containers.

Guaranteed QoS for all workloads — Prevents bursting. Use only for critical databases, not every pod.

Quick Reference

Summary

Set requests based on observed p95 usage. Set memory limits with headroom above peaks. Consider removing CPU limits to avoid throttling. Use LimitRanges for defaults and ResourceQuotas for namespace protection. Monitor throttling and OOMKills as your feedback signal.

---

Frequently Asked Questions

What is the difference between resource requests and limits in Kubernetes?

Requests are what the container is guaranteed and used for scheduling decisions — the scheduler only places pods on nodes with enough allocatable resources. Limits are the maximum a container can use — exceeding memory limits causes OOMKill, exceeding CPU limits causes throttling. Set requests based on normal usage and limits based on peak usage.

How do I determine the right resource requests for my pods?

Monitor actual resource usage over 1-2 weeks using Prometheus metrics or kubectl top pods. Set requests at the P95 usage level and limits at 1.5-2x the request. Tools like Kubernetes VPA (Vertical Pod Autoscaler) can recommend values automatically. Start generous and tighten based on data — under-requesting leads to evictions under pressure.

What happens if I don't set resource limits?

Without limits, a container can consume all available node resources, starving other pods. Without requests, the pod gets BestEffort QoS class and is first to be evicted under memory pressure. Always set at least requests for production workloads. Missing limits can cause noisy-neighbor problems and cascading failures across pods on the same node.

Why is my pod being CPU throttled?

CPU throttling occurs when a container hits its CPU limit — the kernel restricts its CPU time. Check with kubectl top pod or container-level metrics showing throttled_time. Either the CPU limit is too low for the workload, or the application has CPU-intensive bursts. Increase the CPU limit or optimize the code. Note that CPU throttling doesn't kill pods but causes latency spikes.

What are QoS classes in Kubernetes?

Kubernetes assigns Quality of Service classes based on resource settings: Guaranteed (requests = limits for all containers), Burstable (requests set but lower than limits), and BestEffort (no requests or limits). Under memory pressure, BestEffort pods are evicted first, then Burstable, then Guaranteed. Set equal requests and limits for critical production services.

---

Related Resources

• kubectl Cheatsheet — 71 kubectl commands searchable by task
• DevOps Interview Academy — 60 DevOps interview questions including Kubernetes
• Production Troubleshooting Scenarios — 15 production troubleshooting scenarios
• Kubernetes Pod Troubleshooting Guide — Debugging OOMKilled and resource-related failures
• Monitoring and Alerting in Production — Alerting on resource utilization thresholds