TL;DR — Quick Fix
If you're seeing 502s during traffic spikes, apply this ConfigMap immediately:
# Patch NGINX Ingress ConfigMap with production-ready settings
kubectl patch configmap ingress-nginx-controller -n ingress-nginx --type merge -p '{
"data": {
"worker-processes": "auto",
"max-worker-connections": "65536",
"keep-alive": "75",
"keep-alive-requests": "1000",
"upstream-keepalive-connections": "320",
"proxy-read-timeout": "60",
"proxy-send-timeout": "60",
"proxy-buffer-size": "16k",
"proxy-buffers-number": "4"
}
}'
# Restart to pick up changes
kubectl rollout restart deployment ingress-nginx-controller -n ingress-nginx
---
Why NGINX Ingress Returns 502 Errors
A 502 Bad Gateway means NGINX received an invalid response from the upstream. This happens when pods terminate, connections exhaust, or buffers overflow.
---
Worker Connection Tuning
# ingress-nginx-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx
data:
worker-processes: "auto"
max-worker-connections: "65536"
upstream-keepalive-connections: "320"
upstream-keepalive-timeout: "60"
upstream-keepalive-requests: "10000"
---
Keepalive and Timeout Configuration
# Production-ready timeout settings
apiVersion: v1
kind: ConfigMap
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx
data:
keep-alive: "75"
keep-alive-requests: "1000"
client-header-timeout: "60"
client-body-timeout: "60"
proxy-connect-timeout: "5"
proxy-read-timeout: "60"
proxy-send-timeout: "60"
proxy-buffer-size: "16k"
proxy-buffers-number: "4"
large-client-header-buffers: "4 16k"
---
Graceful Connection Draining
The most common cause of 502s during deployments — pods terminate before connections drain:
# deployment.yaml — lifecycle hooks for graceful shutdown
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
template:
spec:
terminationGracePeriodSeconds: 60
containers:
- name: app
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 15"]
readinessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 3
# Verify graceful shutdown — watch for 502s during rollout
kubectl rollout restart deployment my-app
while true; do
STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://my-app.example.com/health)
echo "$(date): $STATUS"
[[ "$STATUS" == "502" ]] && echo "502 DETECTED!"
sleep 0.5
done
---
HPA for Ingress Controller
# ingress-hpa.yaml — Scale controller pods with traffic
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: ingress-nginx-controller
minReplicas: 3
maxReplicas: 15
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 60
- type: Pods
pods:
metric:
name: nginx_ingress_controller_nginx_process_connections
target:
type: AverageValue
averageValue: "5000"
behavior:
scaleUp:
stabilizationWindowSeconds: 30
policies:
- type: Percent
value: 50
periodSeconds: 60
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Pods
value: 1
periodSeconds: 120
---
Real-World Debugging Steps
# Step 1: Check NGINX error logs for 502 patterns
kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx \
--tail=100 | grep -E "502|upstream"
# Step 2: Verify backend endpoints are healthy
kubectl get endpoints my-app-service -o yaml | grep -c "ip:"
# Step 3: Check active connections on the ingress
kubectl exec -n ingress-nginx deploy/ingress-nginx-controller -- \
curl -s localhost:10254/nginx_status
# Step 4: Look for connection resets in config
kubectl exec -n ingress-nginx deploy/ingress-nginx-controller -- \
cat /etc/nginx/nginx.conf | grep -A5 "upstream"
# Annotation-level overrides per Ingress resource
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: api-ingress
annotations:
nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
nginx.ingress.kubernetes.io/proxy-send-timeout: "120"
nginx.ingress.kubernetes.io/proxy-buffer-size: "32k"
nginx.ingress.kubernetes.io/proxy-body-size: "50m"
spec:
ingressClassName: nginx
rules:
- host: api.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api-service
port:
number: 80
---
FAQ
Q: Why do I get 502s only during deployments?
A: Kubernetes removes pod IPs from endpoints before the pod finishes draining connections. Add a preStop hook with sleep 15 to give the ingress time to stop routing.
Q: How many ingress controller replicas should I run?
A: Minimum 3 for HA across AZs. Use HPA to scale based on connections or CPU. Each pod handles roughly 10k-30k concurrent connections depending on instance size.
Q: Should I use proxy-protocol with AWS NLB?
A: Yes, if you need real client IPs. Enable proxy-protocol v2 on both NLB target group and NGINX ConfigMap (use-proxy-protocol: "true").
Q: What's the difference between keep-alive and upstream-keepalive?
A: keep-alive is client-to-NGINX connection reuse. upstream-keepalive-connections is NGINX-to-backend pod pooling. Both reduce latency by avoiding TCP handshake overhead.
Q: How do I handle WebSocket connections through ingress?
A: Add annotations with high timeouts: proxy-read-timeout: "3600" and proxy-send-timeout: "3600". Also set websocket-services annotation.
---