TL;DR — Quick Fix
If pods are stuck in ContainerCreating with failed to assign an IP address errors:
# Check current IP allocation per node
kubectl get nodes -o json | jq '.items[] | {
name: .metadata.name,
allocatable_pods: .status.allocatable.pods
}'
# Enable prefix delegation for 16x more IPs per ENI slot
kubectl set env daemonset aws-node -n kube-system \
ENABLE_PREFIX_DELEGATION=true \
WARM_PREFIX_TARGET=1
# Verify the change is rolling out
kubectl rollout status daemonset/aws-node -n kube-system
This immediately increases IP capacity from ~1 IP per ENI slot to 16 IPs per slot using /28 prefix delegation.
---
Understanding AWS VPC CNI IP Management
The AWS VPC CNI assigns real VPC IP addresses to every pod. Each EC2 instance has a limited number of ENIs and IPs per ENI based on instance type.
---
Subnet Sizing Calculations
Before enabling prefix delegation, verify your subnets have capacity:
#!/bin/bash
# scripts/check-subnet-capacity.sh
echo "=== Subnet IP Capacity Report ==="
SUBNETS=$(aws ec2 describe-subnets \
--filters "Name=tag:kubernetes.io/role/internal-elb,Values=1" \
--query 'Subnets[*].[SubnetId,CidrBlock,AvailableIpAddressCount,AvailabilityZone]' \
--output text)
while IFS=$'\t' read -r subnet_id cidr available az; do
total=$(echo "$cidr" | awk -F/ '{print 2^(32-$2) - 5}')
used=$((total - available))
percent_used=$((used * 100 / total))
status="OK"
[[ $percent_used -gt 70 ]] && status="WARNING"
[[ $percent_used -gt 90 ]] && status="CRITICAL"
echo "${status} ${az} | ${subnet_id} | ${cidr} | Used: ${used}/${total} (${percent_used}%)"
done <<< "$SUBNETS"
Recommended Subnet Sizing
| Cluster Size | Pods per Node | Subnet CIDR | Available IPs | Headroom |
|---|---|---|---|---|
| Small (20 nodes) | 30 | /22 (1019 IPs) | 1019 | ~40% |
| Medium (50 nodes) | 50 | /20 (4091 IPs) | 4091 | ~50% |
| Large (200 nodes) | 110 | /18 (16379 IPs) | 16379 | ~60% |
---
Enabling Prefix Delegation
# aws-node-config.yaml — VPC CNI DaemonSet configuration
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: aws-node
namespace: kube-system
spec:
template:
spec:
containers:
- name: aws-node
env:
- name: ENABLE_PREFIX_DELEGATION
value: "true"
- name: WARM_PREFIX_TARGET
value: "1"
- name: MINIMUM_IP_TARGET
value: "5"
- name: WARM_IP_TARGET
value: "0"
- name: WARM_ENI_TARGET
value: "0"
# Apply and perform rolling restart of nodes
kubectl apply -f aws-node-config.yaml
for node in $(kubectl get nodes -o name); do
kubectl cordon "$node"
kubectl drain "$node" --ignore-daemonsets --delete-emptydir-data --grace-period=120
INSTANCE_ID=$(kubectl get "$node" -o jsonpath='{.spec.providerID}' | cut -d/ -f5)
aws ec2 terminate-instances --instance-ids "$INSTANCE_ID"
sleep 120
kubectl wait --for=condition=Ready nodes --all --timeout=300s
done
---
Custom Networking Mode
For clusters where pod and node IPs must be in different subnets:
# Enable custom networking
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: aws-node
namespace: kube-system
spec:
template:
spec:
containers:
- name: aws-node
env:
- name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
value: "true"
- name: ENI_CONFIG_LABEL_DEF
value: "topology.kubernetes.io/zone"
# ENIConfig per availability zone
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
name: us-east-1a
spec:
securityGroups:
- sg-0123456789abcdef0
subnet: subnet-0abc123pod1a
---
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
name: us-east-1b
spec:
securityGroups:
- sg-0123456789abcdef0
subnet: subnet-0abc123pod1b
---
Monitoring IP Usage
# prometheus-rules/cni-ip-alerts.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: vpc-cni-ip-alerts
spec:
groups:
- name: vpc-cni
rules:
- alert: NodeIPCapacityLow
expr: |
awscni_assigned_ip_addresses / awscni_total_ip_addresses > 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "Node {{ $labels.node }} IP capacity above 80%"
- alert: SubnetIPsExhausted
expr: |
aws_subnet_available_ips < 50
for: 2m
labels:
severity: critical
annotations:
summary: "Subnet {{ $labels.subnet_id }} has fewer than 50 IPs"
# Quick diagnostic for IP exhaustion
kubectl get pods -A --field-selector=status.phase!=Running | grep -i "ContainerCreating\|Pending"
# Check CNI logs for allocation failures
kubectl logs -n kube-system -l k8s-app=aws-node --tail=50 | grep -i "failed\|error\|exhausted"
---
FAQ
Q: How do I know if I'm hitting IP exhaustion vs other scheduling issues?
A: Check pod events with kubectl describe pod <name>. IP exhaustion shows failed to assign an IP address to the container. Also check kubectl logs -n kube-system -l k8s-app=aws-node for ipamd errors.
Q: Can I enable prefix delegation without downtime?
A: You can enable it on the DaemonSet without downtime, but existing nodes won't benefit until their aws-node pod restarts. New nodes use it immediately. For existing nodes, rolling replacement is needed.
Q: What's the difference between custom networking and prefix delegation?
A: Prefix delegation multiplies IPs per ENI slot (1 to 16). Custom networking routes pod IPs through different subnets. You can combine both for subnet isolation plus density.
Q: Will prefix delegation increase my AWS costs?
A: No direct cost increase. IPs within a prefix are free. Running fewer, larger nodes can actually reduce costs compared to many small nodes.
Q: What instance types work best with prefix delegation?
A: Nitro-based instances (m5, c5, r5, m6i) all support it. Older types (t2, m4) do not. Check AWS docs for ENI limits per instance type.
---