AWS EKS in Production — Networking, IAM & Autoscaling Mistakes That Cost Real Money

Introduction

Amazon Elastic Kubernetes Service (EKS) is the managed Kubernetes offering from AWS that removes the operational burden of running your own control plane. But "managed" doesn't mean "zero effort." Running EKS in production requires deliberate decisions around networking, IAM, autoscaling, observability, and cost management.

This guide walks through the full lifecycle of a production EKS deployment — from VPC architecture to CI/CD integration — with real Terraform code and Kubernetes manifests you can adapt for your environment.

---

1. EKS Architecture Overview

EKS separates concerns into two layers:

Control Plane (AWS-managed):

• Runs the Kubernetes API server, etcd, scheduler, and controller manager
• Distributed across multiple Availability Zones automatically
• AWS handles patching, upgrades, and high availability
• You never SSH into control plane nodes

Data Plane (customer-managed):

• Worker nodes where your pods actually run
• You choose the compute type: EC2 instances, Fargate, or a mix
• You control the AMI, instance type, and scaling behavior

Managed Node Groups vs Self-Managed Nodes:

For most production workloads, managed node groups strike the right balance between control and operational simplicity.

---

2. VPC and Networking Design

A production EKS cluster needs a well-planned VPC. The standard pattern uses public and private subnets across at least two (ideally three) Availability Zones.

Architecture:

• Public subnets: Host NAT Gateways and load balancers (ALB/NLB)
• Private subnets: Host worker nodes and pods — no direct internet exposure
• NAT Gateway: Allows private nodes to pull images and reach AWS APIs
• VPC Endpoints (optional): Reduce NAT costs for ECR, S3, STS, and CloudWatch

Pod Networking with VPC CNI:

EKS uses the Amazon VPC CNI plugin by default. Each pod gets a real VPC IP address from the subnet CIDR, enabling direct communication with other AWS services without NAT or overlays.

Key considerations:

• Each EC2 instance has a maximum number of ENIs and IPs per ENI — this limits pod density
• Use ENABLE_PREFIX_DELEGATION to assign /28 prefixes instead of individual IPs (increases pod density by ~4x)
• For large clusters, plan your CIDR blocks carefully or use secondary CIDRs

# Enable prefix delegation on the VPC CNI
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
kubectl set env daemonset aws-node -n kube-system WARM_PREFIX_TARGET=1

Subnet tagging requirements for EKS:

# Private subnets (for internal load balancers and nodes)
kubernetes.io/role/internal-elb = 1
kubernetes.io/cluster/<cluster-name> = shared

# Public subnets (for internet-facing load balancers)
kubernetes.io/role/elb = 1
kubernetes.io/cluster/<cluster-name> = shared

---

3. Node Groups: Managed, Self-Managed, and Fargate

Managed Node Groups (recommended for most workloads):

• AWS manages the underlying Auto Scaling Group
• Automatic draining during upgrades
• Supports custom launch templates for userdata, AMI, and block device config
• Use for stateless microservices, web apps, and general workloads

Self-Managed Node Groups:

• Full control over the ASG, launch template, and lifecycle hooks
• Required when you need custom AMIs with pre-baked software
• Use for GPU workloads, Windows nodes, or compliance-heavy environments

Fargate Profiles:

• Serverless — no nodes to manage at all
• Each pod runs in its own micro-VM (Firecracker)
• No DaemonSets, no SSH access, limited to 4 vCPU / 30 GB RAM per pod
• Use for batch jobs, cron workloads, or low-traffic services where you want zero node management

# Example Fargate profile definition
apiVersion: eks.amazonaws.com/v1
kind: FargateProfile
metadata:
  name: batch-workloads
spec:
  selectors:
    - namespace: batch
      labels:
        compute: fargate
  subnetIds:
    - subnet-0123456789abcdef0
    - subnet-0123456789abcdef1

Decision Framework:

• Default to managed node groups for general workloads
• Use Fargate for bursty, short-lived, or batch workloads
• Use self-managed only when you need deep OS-level customization

---

4. IAM Integration: IRSA (IAM Roles for Service Accounts)

IRSA is the recommended way to grant AWS permissions to pods. It replaces the older approach of attaching IAM roles to nodes (which gives every pod on that node the same permissions).

How IRSA works:

EKS cluster has an OIDC provider

You create an IAM role with a trust policy that references the OIDC provider and a specific Kubernetes service account

Pods using that service account automatically receive temporary AWS credentials via the projected service account token

# Associate OIDC provider with your cluster
eksctl utils associate-iam-oidc-provider \
  --cluster my-cluster \
  --approve

IAM Role Trust Policy for IRSA:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:my-namespace:my-service-account",
          "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

Annotate the Kubernetes Service Account:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account
  namespace: my-namespace
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/my-irsa-role

Best Practices for IRSA:

• One IAM role per microservice — never share roles across unrelated services
• Use condition keys to restrict to specific service accounts
• Audit role usage with CloudTrail
• Prefer IRSA over kube2iam or kiam (they rely on instance metadata interception)

---

5. Cluster Autoscaler and Karpenter

Cluster Autoscaler is the traditional approach — it watches for pending pods that can't be scheduled and scales up the Auto Scaling Group.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels:
        app: cluster-autoscaler
    spec:
      serviceAccountName: cluster-autoscaler
      containers:
        - name: cluster-autoscaler
          image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.29.0
          command:
            - ./cluster-autoscaler
            - --v=4
            - --stderrthreshold=info
            - --cloud-provider=aws
            - --skip-nodes-with-local-storage=false
            - --expander=least-waste
            - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/my-cluster

Karpenter (recommended for new clusters):

Karpenter is AWS's next-generation autoscaler that provisions nodes directly (bypassing ASGs). It's faster, more flexible, and cost-aware by default.

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["4"]
      nodeClassRef:
        name: default
  limits:
    cpu: "1000"
    memory: 1000Gi
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h

Karpenter vs Cluster Autoscaler:

---

6. ALB Ingress Controller Setup

The AWS Load Balancer Controller provisions Application Load Balancers (ALB) and Network Load Balancers (NLB) based on Kubernetes Ingress and Service resources.

Installation via Helm:

# Add the EKS Helm chart repo
helm repo add eks https://aws.github.io/eks-charts
helm repo update

# Install the AWS Load Balancer Controller
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=my-cluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller

Ingress Resource Example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-ingress
  namespace: production
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:111122223333:certificate/abc123
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/healthcheck-path: /health
spec:
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-app-service
                port:
                  number: 80

Key ALB annotations:

• target-type: ip — routes directly to pod IPs (required for Fargate, recommended for VPC CNI)
• scheme: internet-facing vs internal — controls public vs private ALB
• ssl-redirect — automatically redirects HTTP to HTTPS
• group.name — share a single ALB across multiple Ingress resources to reduce cost

---

7. Observability: Monitoring, Logging, and Tracing

CloudWatch Container Insights:

The quickest path to EKS observability. Provides CPU, memory, disk, and network metrics at cluster, node, pod, and container level.

# Install CloudWatch agent via Helm
helm repo add aws-observability https://aws-observability.github.io/helm-charts
helm install amazon-cloudwatch aws-observability/amazon-cloudwatch-observability \
  -n amazon-cloudwatch --create-namespace \
  --set clusterName=my-cluster \
  --set region=us-east-1

Prometheus + Grafana Stack (recommended for deeper visibility):

# Install kube-prometheus-stack
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install monitoring prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace \
  --set grafana.adminPassword=secure-password \
  --set prometheus.prometheusSpec.retention=30d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi

Amazon Managed Prometheus (AMP) + Amazon Managed Grafana (AMG):

For teams that want Prometheus/Grafana without managing the infrastructure:

# Prometheus remote write configuration to AMP
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-remote-write
data:
  remote-write.yaml: |
    remote_write:
      - url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-xxxxx/api/v1/remote_write
        sigv4:
          region: us-east-1
        queue_config:
          max_samples_per_send: 1000
          max_shards: 200
          capacity: 2500

Logging with Fluent Bit:

# Install Fluent Bit as DaemonSet for log forwarding
helm install fluent-bit fluent/fluent-bit \
  -n logging --create-namespace \
  --set output.cloudWatch.region=us-east-1 \
  --set output.cloudWatch.logGroupName=/eks/my-cluster \
  --set output.cloudWatch.autoCreateGroup=true

---

8. Security Best Practices

Pod Security Standards:

EKS supports Pod Security Admission (PSA) — the built-in replacement for PodSecurityPolicies:

# Enforce restricted security standard on a namespace
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Network Policies:

Use Calico or VPC CNI network policies to restrict pod-to-pod traffic:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress: []
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

Secrets Management:

• Use AWS Secrets Manager or Parameter Store with the Secrets Store CSI Driver
• Never store secrets in ConfigMaps or environment variables in plain text
• Enable envelope encryption for Kubernetes secrets with a KMS key

# Enable secrets encryption on the EKS cluster
aws eks create-cluster \
  --name my-cluster \
  --encryption-config '[{"resources":["secrets"],"provider":{"keyArn":"arn:aws:kms:us-east-1:111122223333:key/key-id"}}]'

Additional Security Hardening:

• Disable public API endpoint or restrict CIDR access
• Enable audit logging to CloudWatch
• Use GuardDuty for EKS runtime threat detection
• Regularly scan container images with ECR image scanning or Trivy
• Implement OPA Gatekeeper or Kyverno for policy enforcement

---

9. CI/CD Integration: Deploying to EKS

GitHub Actions Example:

name: Deploy to EKS
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/github-actions-eks-deploy
          aws-region: us-east-1

      - name: Login to ECR
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build and push image
        run: |
          docker build -t $ECR_REGISTRY/my-app:$GITHUB_SHA .
          docker push $ECR_REGISTRY/my-app:$GITHUB_SHA

      - name: Update kubeconfig
        run: aws eks update-kubeconfig --name my-cluster --region us-east-1

      - name: Deploy with kubectl
        run: |
          kubectl set image deployment/my-app \
            my-app=$ECR_REGISTRY/my-app:$GITHUB_SHA \
            -n production
          kubectl rollout status deployment/my-app -n production --timeout=300s

Jenkins Pipeline Example:

pipeline {
    agent any
    environment {
        AWS_REGION = 'us-east-1'
        CLUSTER_NAME = 'my-cluster'
        ECR_REPO = '111122223333.dkr.ecr.us-east-1.amazonaws.com/my-app'
    }
    stages {
        stage('Build') {
            steps {
                sh "docker build -t ${ECR_REPO}:${BUILD_NUMBER} ."
            }
        }
        stage('Push to ECR') {
            steps {
                sh "aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${ECR_REPO}"
                sh "docker push ${ECR_REPO}:${BUILD_NUMBER}"
            }
        }
        stage('Deploy to EKS') {
            steps {
                sh "aws eks update-kubeconfig --name ${CLUSTER_NAME} --region ${AWS_REGION}"
                sh "kubectl set image deployment/my-app my-app=${ECR_REPO}:${BUILD_NUMBER} -n production"
                sh "kubectl rollout status deployment/my-app -n production --timeout=300s"
            }
        }
    }
}

Best Practices for CI/CD to EKS:

• Use OIDC federation for GitHub Actions (no long-lived credentials)
• Implement GitOps with ArgoCD or Flux for declarative deployments
• Use Helm or Kustomize for environment-specific configuration
• Add canary or blue-green deployment strategies with Argo Rollouts
• Gate deployments with automated integration tests

---

10. Cost Optimization Strategies

Spot Instances:

Spot instances can save 60-90% over On-Demand pricing. Use them for stateless, fault-tolerant workloads.

# Karpenter NodePool with Spot priority
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: spot-workloads
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-size
          operator: In
          values: ["large", "xlarge", "2xlarge"]
  disruption:
    consolidationPolicy: WhenUnderutilized

Right-Sizing with VPA:

Vertical Pod Autoscaler recommends (or automatically sets) resource requests based on actual usage:

# Install VPA
helm install vpa fairwinds-stable/vpa \
  -n vpa --create-namespace \
  --set recommender.enabled=true \
  --set updater.enabled=false  # Start with recommendations only

Karpenter Consolidation:

Karpenter actively consolidates workloads onto fewer, cheaper nodes:

• Replaces underutilized nodes with smaller instances
• Moves workloads to Spot instances when safe
• Respects PodDisruptionBudgets during consolidation

Additional Cost Strategies:

• Use Savings Plans or Reserved Instances for baseline On-Demand capacity
• Implement resource quotas per namespace to prevent waste
• Schedule non-production clusters to scale down during off-hours
• Use EKS with Fargate for sporadic workloads (pay-per-pod, no idle nodes)
• Monitor costs with Kubecost or AWS Cost Explorer with EKS cost allocation tags
• Share ALBs across services using IngressGroup annotations

Cost Comparison (typical 50-node cluster):

---

11. Terraform Example: Complete EKS Cluster

# providers.tf
terraform {
  required_version = ">= 1.5"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.25"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.12"
    }
  }
}

provider "aws" {
  region = var.region
}

# variables.tf
variable "region" {
  default = "us-east-1"
}

variable "cluster_name" {
  default = "production-eks"
}

variable "cluster_version" {
  default = "1.29"
}

# vpc.tf — Using the official VPC module
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.cluster_name}-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["${var.region}a", "${var.region}b", "${var.region}c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = false
  enable_dns_hostnames = true

  public_subnet_tags = {
    "kubernetes.io/role/elb"                      = 1
    "kubernetes.io/cluster/${var.cluster_name}"    = "shared"
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb"             = 1
    "kubernetes.io/cluster/${var.cluster_name}"    = "shared"
    "karpenter.sh/discovery"                      = var.cluster_name
  }
}

# eks.tf — Using the official EKS module
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = var.cluster_name
  cluster_version = var.cluster_version

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  cluster_endpoint_public_access  = true
  cluster_endpoint_private_access = true

  # Enable IRSA
  enable_irsa = true

  # Encrypt secrets with KMS
  cluster_encryption_config = {
    resources = ["secrets"]
  }

  # Managed Node Groups
  eks_managed_node_groups = {
    system = {
      instance_types = ["m6i.large"]
      min_size       = 2
      max_size       = 4
      desired_size   = 2

      labels = {
        role = "system"
      }

      taints = []
    }

    application = {
      instance_types = ["m6i.xlarge", "m6a.xlarge", "m5.xlarge"]
      min_size       = 2
      max_size       = 20
      desired_size   = 3
      capacity_type  = "SPOT"

      labels = {
        role = "application"
      }
    }
  }

  # Cluster access management
  enable_cluster_creator_admin_permissions = true

  tags = {
    Environment = "production"
    Terraform   = "true"
  }
}

# outputs.tf
output "cluster_endpoint" {
  value = module.eks.cluster_endpoint
}

output "cluster_certificate_authority_data" {
  value = module.eks.cluster_certificate_authority_data
}

output "cluster_name" {
  value = module.eks.cluster_name
}

Deploy the cluster:

terraform init
terraform plan -out=tfplan
terraform apply tfplan

# Update kubeconfig
aws eks update-kubeconfig --name production-eks --region us-east-1

---

12. EKS vs ECS vs Fargate: Decision Matrix

When to choose EKS:

• You need Kubernetes-specific features (CRDs, operators, Helm ecosystem)
• Your team already knows Kubernetes
• You want multi-cloud portability or hybrid deployments
• You run complex stateful workloads (databases, message queues)

When to choose ECS:

• You want simpler operations with deep AWS integration
• Your team is small and doesn't want to learn Kubernetes
• Your workloads are straightforward (web apps, APIs, workers)
• You prioritize fast time-to-production over flexibility

When to choose Fargate (with either EKS or ECS):

• You want zero node management
• Your workloads are bursty or unpredictable
• You're running batch jobs, cron tasks, or dev/test environments
• You want to pay only when code is running

---

Conclusion

Running EKS in production is a journey, not a destination. Start with the fundamentals — proper VPC design, IRSA for security, and managed node groups for simplicity. As your platform matures, layer in Karpenter for intelligent autoscaling, implement GitOps for deployment consistency, and optimize costs with Spot instances and consolidation.

The key principles:

Security first — IRSA, network policies, pod security standards, and secrets encryption from day one

Observe everything — you can't fix what you can't see

Automate scaling — Karpenter removes the manual capacity planning burden

Optimize continuously — right-size resources, use Spot where safe, and consolidate aggressively

Keep it simple — start with managed services and only take on complexity when you outgrow them

EKS gives you the full power of Kubernetes with AWS handling the undifferentiated heavy lifting of the control plane. Your job is to make smart decisions about everything above the API server.

---

Frequently Asked Questions

What is AWS EKS and when should I use it?

AWS EKS (Elastic Kubernetes Service) is a managed Kubernetes control plane that handles master node availability, patching, and upgrades. Use EKS when your team has Kubernetes expertise, needs multi-cloud portability, or runs complex microservice architectures. For simpler container workloads, ECS may be more cost-effective with less operational overhead.

How much does EKS cost in production?

EKS charges $0.10 per hour ($73/month) for the control plane, plus your worker node costs (EC2 instances or Fargate). A typical production setup with 3 m5.large worker nodes costs approximately $250-350/month total. Add costs for load balancers, storage, and data transfer based on your traffic patterns.

How do I upgrade an EKS cluster without downtime?

Use a rolling update strategy by first upgrading the control plane through the AWS console or eksctl, then updating worker nodes using managed node groups with a rolling update policy. Set PodDisruptionBudgets on critical workloads and perform the upgrade during low-traffic periods. Always test the upgrade path in a staging cluster first.

What is the difference between EKS managed node groups and self-managed nodes?

Managed node groups automate node provisioning, OS patching, and graceful draining during updates, reducing operational burden. Self-managed nodes give you full control over the AMI, instance type, and lifecycle but require you to handle all upgrades manually. Use managed node groups unless you need custom AMIs or specialized instance configurations.

How do I set up cluster autoscaling on EKS?

Deploy the Kubernetes Cluster Autoscaler or Karpenter to automatically adjust node count based on pending pod requests. Karpenter is recommended for new deployments as it provisions right-sized instances faster and supports diverse instance types. Configure appropriate min/max node counts and ensure your IAM roles allow EC2 scaling operations.