TL;DR — Quick Fix
Survive spot interruptions with three essentials: Node Termination Handler, Pod Disruption Budgets, and graceful shutdown hooks.
# Install AWS Node Termination Handler via Helm
helm repo add eks https://aws.github.io/eks-charts
helm install aws-node-termination-handler eks/aws-node-termination-handler \
--namespace kube-system \
--set enableSpotInterruptionDraining=true \
--set enableScheduledEventDraining=true \
--set enableRebalanceMonitoring=true
# Pod Disruption Budget — never let all replicas die
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: my-app-pdb
spec:
minAvailable: "50%"
selector:
matchLabels:
app: my-app
---
Architecture — Spot Instance Lifecycle in Kubernetes
---
Step 1 — Configure Karpenter for Spot with On-Demand Fallback
# karpenter-nodepool.yaml
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
name: spot-pool
spec:
template:
spec:
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["spot", "on-demand"]
- key: kubernetes.io/arch
operator: In
values: ["amd64"]
- key: karpenter.k8s.aws/instance-category
operator: In
values: ["m", "c", "r"]
- key: karpenter.k8s.aws/instance-size
operator: In
values: ["large", "xlarge", "2xlarge"]
nodeClassRef:
name: default
disruption:
consolidationPolicy: WhenUnderutilized
expireAfter: 720h
limits:
cpu: "1000"
memory: 1000Gi
weight: 80
---
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
name: on-demand-critical
spec:
template:
spec:
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["on-demand"]
- key: kubernetes.io/arch
operator: In
values: ["amd64"]
nodeClassRef:
name: default
taints:
- key: workload-type
value: critical
effect: NoSchedule
limits:
cpu: "200"
memory: 200Gi
weight: 20
---
Step 2 — Pod Disruption Budgets
# pdb-stateless-app.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: web-frontend-pdb
spec:
minAvailable: "60%"
selector:
matchLabels:
app: web-frontend
---
# pdb-stateful-app.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: redis-cluster-pdb
spec:
maxUnavailable: 1
selector:
matchLabels:
app: redis-cluster
---
Step 3 — Graceful Shutdown Lifecycle Hooks
# deployment with graceful shutdown
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-server
spec:
replicas: 4
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template:
spec:
terminationGracePeriodSeconds: 90
containers:
- name: api
image: myapp:latest
ports:
- containerPort: 8080
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 5 && kill -SIGTERM 1"]
readinessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: api-server
---
Step 4 — Node Termination Handler (Queue Mode)
Queue mode is recommended for production — it uses SQS instead of IMDS polling:
# Create SQS queue and EventBridge rules
aws sqs create-queue --queue-name spot-interruption-queue
# EventBridge rule for spot interruptions
aws events put-rule \
--name spot-interruption-rule \
--event-pattern '{
"source": ["aws.ec2"],
"detail-type": [
"EC2 Spot Instance Interruption Warning",
"EC2 Instance Rebalance Recommendation"
]
}'
aws events put-targets --rule spot-interruption-rule \
--targets "Id"="sqs-target","Arn"="arn:aws:sqs:us-east-1:123456789:spot-interruption-queue"
# Helm values for queue mode
aws-node-termination-handler:
enableSqsTerminationDraining: true
queueURL: "https://sqs.us-east-1.amazonaws.com/123456789/spot-interruption-queue"
enableSpotInterruptionDraining: true
enableRebalanceMonitoring: true
enableScheduledEventDraining: true
podTerminationGracePeriod: 60
nodeTerminationGracePeriod: 90
taintNode: true
---
Step 5 — Cost Savings Calculator
#!/bin/bash
# spot-savings-report.sh — Calculate monthly savings
ON_DEMAND_RATE=0.192 # m5.xlarge us-east-1
SPOT_RATE=0.058 # average spot price
NODES=20
HOURS_PER_MONTH=730
on_demand_cost=$(echo "$ON_DEMAND_RATE $NODES $HOURS_PER_MONTH" | bc)
spot_cost=$(echo "$SPOT_RATE $NODES $HOURS_PER_MONTH" | bc)
savings=$(echo "$on_demand_cost - $spot_cost" | bc)
percentage=$(echo "scale=0; ($savings / $on_demand_cost) * 100" | bc)
echo "Monthly Cost Report:"
echo " On-Demand: \$$on_demand_cost"
echo " Spot: \$$spot_cost"
echo " Savings: \$$savings (${percentage}%)"
---
Frequently Asked Questions
What happens if Karpenter can't find spot capacity?
Karpenter automatically falls back to on-demand instances when spot capacity is unavailable, as long as your NodePool includes both spot and on-demand in capacity-type. The fallback is transparent — pods are scheduled normally on on-demand nodes.
Should I use Node Termination Handler in IMDS or Queue mode?
Queue mode (SQS) is recommended for production. IMDS mode requires a DaemonSet polling every node's metadata endpoint, adding overhead. Queue mode uses a single Deployment that processes centralized SQS events, is more reliable, and catches events that IMDS misses.
How do I prevent stateful workloads from landing on spot nodes?
Use taints on spot nodes and tolerations only on stateless pods. Alternatively, use nodeAffinity to pin stateful workloads (databases, Kafka) to on-demand nodes, while allowing stateless services to run on spot.
What's the minimum replica count for zero-downtime on spot?
At least 3 replicas spread across 2+ availability zones. With a PDB of minAvailable: 50%, Kubernetes will only drain pods if at least half remain running. Combined with topologySpreadConstraints, you survive single-AZ spot reclaims.
Can I use spot instances for batch/cron jobs?
Yes, and it's ideal. Batch jobs are inherently interruption-tolerant. Use Kubernetes Jobs with restartPolicy: OnFailure so interrupted jobs resume automatically. Set backoffLimit: 3 and use checkpointing for long-running batch processing.
---