TL;DR Quick Fix
Safe node drain in three commands:
# 1. Cordon the node (prevent new pod scheduling)
kubectl cordon worker-node-3
# 2. Drain with PDB respect and grace period
kubectl drain worker-node-3 \
--ignore-daemonsets \
--delete-emptydir-data \
--grace-period=120 \
--timeout=300s
# 3. After maintenance, uncordon
kubectl uncordon worker-node-3
If you have pods with local storage that cannot be rescheduled, add --delete-emptydir-data. If drain hangs, check for pods without PDBs or stuck finalizers.
---
Architecture Overview
---
Understanding Cordon vs. Drain
Cordon — Mark Node as Unschedulable
# Cordon prevents NEW pods from being scheduled, existing pods keep running
kubectl cordon worker-node-3
# Verify the node is cordoned
kubectl get nodes
# NAME STATUS ROLES AGE
# worker-node-3 Ready,SchedulingDisabled <none> 45d
Drain — Evict All Pods from Node
# Full drain command with safety options
kubectl drain worker-node-3 \
--ignore-daemonsets \
--delete-emptydir-data \
--grace-period=120 \
--timeout=300s \
--pod-selector='app!=critical-singleton'
# Force drain when PDBs block (use with extreme caution)
kubectl drain worker-node-3 \
--ignore-daemonsets \
--delete-emptydir-data \
--force \
--disable-eviction
---
Pod Disruption Budgets (PDBs)
PDBs are the safety net that prevents drain operations from taking down too many replicas at once.
PDB Configuration Examples
# Percentage-based PDB
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: web-app-pdb
namespace: production
spec:
minAvailable: "80%"
selector:
matchLabels:
app: web-app
---
# Count-based PDB for stateful services
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: redis-pdb
namespace: production
spec:
maxUnavailable: 1
selector:
matchLabels:
app: redis-cluster
Check PDB Status Before Draining
#!/bin/bash
# check-pdb-status.sh — verify PDBs allow disruption
set -euo pipefail
NODE=$1
echo "Checking PDB status for pods on node: $NODE"
PODS_ON_NODE=$(kubectl get pods --all-namespaces \
--field-selector spec.nodeName=$NODE \
-o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}')
kubectl get pdb --all-namespaces -o wide
echo ""
echo "Pods on $NODE that are protected by PDBs:"
for pod in $PODS_ON_NODE; do
NS=$(echo $pod | cut -d'/' -f1)
POD_NAME=$(echo $pod | cut -d'/' -f2)
LABELS=$(kubectl get pod -n $NS $POD_NAME -o jsonpath='{.metadata.labels}')
echo " $NS/$POD_NAME"
done
---
Handling DaemonSets and Local Storage
# DaemonSets are ignored during drain (they run on all nodes)
kubectl drain worker-node-3 --ignore-daemonsets
# Pods with emptyDir volumes — data will be lost
kubectl drain worker-node-3 --delete-emptydir-data
# Pods with local PVs — requires special handling
# First, check which pods use local storage
kubectl get pods --field-selector spec.nodeName=worker-node-3 \
-o jsonpath='{range .items[]}{.metadata.name}: {.spec.volumes[].persistentVolumeClaim.claimName}{"\n"}{end}'
# For StatefulSets with local storage, use a pre-drain job to backup data
apiVersion: batch/v1
kind: Job
metadata:
name: backup-local-data
spec:
template:
spec:
nodeSelector:
kubernetes.io/hostname: worker-node-3
containers:
- name: backup
image: alpine:3.19
command: ["/bin/sh", "-c"]
args:
- |
tar czf /backup/data-$(date +%Y%m%d).tar.gz /data
aws s3 cp /backup/data-*.tar.gz s3://backups/node-drain/
volumeMounts:
- name: local-data
mountPath: /data
- name: backup-vol
mountPath: /backup
volumes:
- name: local-data
persistentVolumeClaim:
claimName: app-data-pvc
- name: backup-vol
emptyDir: {}
restartPolicy: Never
---
Automated Node Rotation with Karpenter
# Karpenter NodePool with automated expiry
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
name: default
spec:
template:
spec:
nodeClassRef:
name: default
requirements:
- key: kubernetes.io/arch
operator: In
values: ["amd64"]
- key: karpenter.sh/capacity-type
operator: In
values: ["on-demand"]
disruption:
consolidationPolicy: WhenUnderutilized
expireAfter: 720h # Force node replacement every 30 days
budgets:
- nodes: "10%" # Max 10% of nodes disrupted simultaneously
# Trigger Karpenter drift for AMI updates
# Update the EC2NodeClass with new AMI
kubectl patch ec2nodeclass default --type merge -p '{
"spec": {
"amiSelectorTerms": [
{
"id": "ami-0abcdef1234567890"
}
]
}
}'
# Karpenter will detect drift and rotate nodes automatically
kubectl get nodeclaims -w
---
EKS Managed Node Group Upgrades
#!/bin/bash
# eks-node-upgrade.sh — upgrade EKS managed node groups
set -euo pipefail
CLUSTER_NAME="production-cluster"
NODEGROUP_NAME="workers-general"
TARGET_VERSION="1.29"
echo "Starting EKS node group upgrade..."
# Update the node group AMI version
aws eks update-nodegroup-version \
--cluster-name $CLUSTER_NAME \
--nodegroup-name $NODEGROUP_NAME \
--kubernetes-version $TARGET_VERSION \
--force
# Monitor the upgrade
watch -n 10 "aws eks describe-nodegroup \
--cluster-name $CLUSTER_NAME \
--nodegroup-name $NODEGROUP_NAME \
--query 'nodegroup.{status:status,version:version,health:health}'"
# EKS Node Group with update config for controlled rolling
apiVersion: eks.amazonaws.com/v1
kind: NodeGroup
metadata:
name: workers-general
spec:
updateConfig:
maxUnavailable: 1
# OR use percentage
# maxUnavailablePercentage: 25
launchTemplate:
version: "$Latest"
---
Zero-Downtime Node Replacement Script
#!/bin/bash
# safe-node-replace.sh — full zero-downtime node replacement
set -euo pipefail
OLD_NODE=$1
NEW_NODE_LABEL=${2:-"node-role.kubernetes.io/worker="}
echo "=== Starting safe node replacement for: $OLD_NODE ==="
# Step 1: Verify PDBs are satisfied
echo "[1/5] Checking PDB status..."
BLOCKED_PDBS=$(kubectl get pdb --all-namespaces -o json | \
jq -r '.items[] | select(.status.disruptionsAllowed == 0) | .metadata.name')
if [ -n "$BLOCKED_PDBS" ]; then
echo "WARNING: These PDBs currently block disruption:"
echo "$BLOCKED_PDBS"
echo "Waiting for PDBs to allow disruption..."
sleep 30
fi
# Step 2: Cordon the old node
echo "[2/5] Cordoning $OLD_NODE..."
kubectl cordon "$OLD_NODE"
# Step 3: Wait for replacement capacity
echo "[3/5] Waiting for new node capacity..."
kubectl wait --for=condition=Ready node -l "$NEW_NODE_LABEL" --timeout=300s
# Step 4: Drain with safety checks
echo "[4/5] Draining $OLD_NODE..."
kubectl drain "$OLD_NODE" \
--ignore-daemonsets \
--delete-emptydir-data \
--grace-period=120 \
--timeout=600s
# Step 5: Verify all pods rescheduled successfully
echo "[5/5] Verifying pod health..."
sleep 30
PENDING_PODS=$(kubectl get pods --all-namespaces \
--field-selector status.phase=Pending \
-o jsonpath='{.items[*].metadata.name}')
if [ -n "$PENDING_PODS" ]; then
echo "WARNING: Some pods are still Pending: $PENDING_PODS"
exit 1
fi
echo "=== Node replacement complete ==="
---
FAQ
What happens if drain times out?
The drain command will exit with an error, but the node remains cordoned. Pods that could not be evicted within the timeout are still running. Check for pods stuck in Terminating state (kubectl get pods --field-selector spec.nodeName=<node> | grep Terminating) and investigate stuck finalizers.
Can I drain multiple nodes simultaneously?
Yes, but be careful with PDBs. If you have maxUnavailable: 1 on a PDB and try to drain two nodes hosting that workload, the second drain will block. Use tools like kubectl-drain-controller or Karpenter's disruption budgets to automate safe parallel drains.
How do I handle nodes with critical singleton pods?
Use --pod-selector to exclude critical pods from the drain, or better yet, convert singletons to HA deployments with PDBs. If a singleton must exist, use a pre-drain hook to gracefully migrate it before the drain starts.
What is the difference between --force and --disable-eviction?
--force deletes pods that are not managed by a controller (bare pods). --disable-eviction bypasses the Eviction API entirely, meaning PDBs are ignored. Never use --disable-eviction in production unless you understand the blast radius.
How long should I set the grace period?
Match it to your application's shutdown time. If your app needs 60 seconds to drain connections and finish in-flight requests, set --grace-period=90 (add buffer). Check your pod's terminationGracePeriodSeconds — the drain grace period should be at least equal to that.
---