Zero-Downtime Kubernetes Deployments — Why Rolling Updates Still Drop Requests

Deployments Should Be Boring

The best deployment is the one nobody notices. No downtime, no error spikes, no frantic Slack messages. But achieving that requires understanding the tradeoffs between different strategies and configuring your health checks properly.

I've seen teams lose revenue because a bad deploy took down production for 20 minutes. I've also seen teams deploy 50 times a day without a single user noticing. The difference isn't luck — it's strategy.

Strategy 1: Rolling Updates

The default Kubernetes deployment strategy. Pods are replaced gradually — old pods are terminated as new ones become ready.

When to Use

• Standard deployments where brief mixed-version traffic is acceptable
• Stateless services with backward-compatible changes
• When you don't have infrastructure budget for running duplicate environments

Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
  namespace: production
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2        # At most 2 extra pods during rollout
      maxUnavailable: 0  # Never reduce below desired count
  selector:
    matchLabels:
      app: api-service
  template:
    metadata:
      labels:
        app: api-service
        version: v2.3.1
    spec:
      containers:
        - name: api
          image: myregistry/api-service:v2.3.1
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /health/live
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 10
            failureThreshold: 3
          startupProbe:
            httpGet:
              path: /health/ready
              port: 8080
            initialDelaySeconds: 0
            periodSeconds: 2
            failureThreshold: 30  # 60 seconds to start up
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 10"]
      terminationGracePeriodSeconds: 60

Key settings:

• maxUnavailable: 0 ensures you never drop below your replica count during deploys.
• maxSurge: 2 controls how fast the rollout progresses.
• preStop sleep gives in-flight requests time to complete before the pod is killed.
• startupProbe prevents liveness checks from killing slow-starting pods.

Monitoring a Rolling Deployment

# Watch the rollout
kubectl rollout status deployment/api-service -n production

# Check rollout history
kubectl rollout history deployment/api-service -n production

# Instant rollback if something goes wrong
kubectl rollout undo deployment/api-service -n production

Strategy 2: Blue-Green Deployments

Run two identical environments. "Blue" is live, "green" is the new version. Switch traffic atomically by updating the Service selector.

When to Use

• Database migrations that need the full new deployment running before cutting over
• When you need instant rollback (just flip the selector back)
• Compliance environments where you must validate the full deployment before going live

Implementation With kubectl

# blue deployment (currently serving traffic)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service-blue
  namespace: production
spec:
  replicas: 6
  selector:
    matchLabels:
      app: api-service
      slot: blue
  template:
    metadata:
      labels:
        app: api-service
        slot: blue
        version: v2.3.0
    spec:
      containers:
        - name: api
          image: myregistry/api-service:v2.3.0
---
# green deployment (new version, not receiving traffic yet)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service-green
  namespace: production
spec:
  replicas: 6
  selector:
    matchLabels:
      app: api-service
      slot: green
  template:
    metadata:
      labels:
        app: api-service
        slot: green
        version: v2.3.1
    spec:
      containers:
        - name: api
          image: myregistry/api-service:v2.3.1
---
# Service points to blue (active slot)
apiVersion: v1
kind: Service
metadata:
  name: api-service
  namespace: production
spec:
  selector:
    app: api-service
    slot: blue  # <- Change this to 'green' to switch
  ports:
    - port: 80
      targetPort: 8080

Switching Traffic

# Verify green is healthy
kubectl get pods -l slot=green -n production
kubectl exec -it deploy/api-service-green -n production -- curl localhost:8080/health/ready

# Switch traffic to green
kubectl patch service api-service -n production \
  -p '{"spec":{"selector":{"slot":"green"}}}'

# Verify traffic is flowing to green
kubectl get endpoints api-service -n production

# Rollback if needed (switch back to blue)
kubectl patch service api-service -n production \
  -p '{"spec":{"selector":{"slot":"blue"}}}'

# Once stable, scale down blue
kubectl scale deployment api-service-blue --replicas=0 -n production

Blue-Green With ArgoCD

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api-service
  namespace: production
spec:
  replicas: 6
  strategy:
    blueGreen:
      activeService: api-service-active
      previewService: api-service-preview
      autoPromotionEnabled: false
      prePromotionAnalysis:
        templates:
          - templateName: smoke-tests
        args:
          - name: service-name
            value: api-service-preview
      scaleDownDelaySeconds: 600  # Keep old version for 10 min after switch
  selector:
    matchLabels:
      app: api-service
  template:
    metadata:
      labels:
        app: api-service
    spec:
      containers:
        - name: api
          image: myregistry/api-service:v2.3.1

Strategy 3: Canary Deployments

Route a small percentage of traffic to the new version. Monitor for errors. Gradually increase if healthy. Roll back immediately if not.

When to Use

• High-traffic services where even brief errors affect many users
• When you want data-driven confidence before full rollout
• Changes with uncertain impact (performance changes, new algorithms)

Canary With Argo Rollouts

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api-service
  namespace: production
spec:
  replicas: 10
  strategy:
    canary:
      canaryService: api-service-canary
      stableService: api-service-stable
      trafficRouting:
        istio:
          virtualService:
            name: api-service-vsvc
            routes:
              - primary
      steps:
        - setWeight: 5
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: error-rate-check
            args:
              - name: service-name
                value: api-service-canary
        - setWeight: 20
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: error-rate-check
              - templateName: latency-check
        - setWeight: 50
        - pause: { duration: 10m }
        - analysis:
            templates:
              - templateName: full-analysis
        - setWeight: 100
  selector:
    matchLabels:
      app: api-service
  template:
    metadata:
      labels:
        app: api-service
    spec:
      containers:
        - name: api
          image: myregistry/api-service:v2.3.1

Analysis Template (Auto-Rollback on Errors)

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate-check
spec:
  args:
    - name: service-name
  metrics:
    - name: error-rate
      interval: 1m
      count: 5
      successCondition: result[0] < 0.05
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{status=~"5..",service="{{args.service-name}}"}[2m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[2m]))
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: latency-check
spec:
  args:
    - name: service-name
  metrics:
    - name: p99-latency
      interval: 1m
      count: 5
      successCondition: result[0] < 2.0
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            histogram_quantile(0.99,
              sum(rate(http_request_duration_seconds_bucket{service="{{args.service-name}}"}[2m])) by (le)
            )

Health Checks: The Foundation of Zero-Downtime

None of these strategies work without proper health checks. Kubernetes uses three probes:

Common Health Check Mistakes

Liveness check hits a database. Database goes down → all pods restart → cascading failure. Liveness should only check if the process is stuck, not dependencies.

No startup probe for slow apps. The liveness probe kills pods that haven't finished starting yet, creating restart loops.

readinessProbe too aggressive. 1-second timeout with 1 failure threshold means a single slow GC pause removes pods from load balancing.

Proper Health Check Implementation

# Application code (Go example)
# /health/live - Is the process running and not deadlocked?
# /health/ready - Can this instance serve traffic right now?

startupProbe:
  httpGet:
    path: /health/ready
    port: 8080
  periodSeconds: 2
  failureThreshold: 30     # 60 seconds to start

readinessProbe:
  httpGet:
    path: /health/ready     # Checks DB connection, cache, dependencies
    port: 8080
  periodSeconds: 5
  failureThreshold: 3
  successThreshold: 1

livenessProbe:
  httpGet:
    path: /health/live      # Only checks process health, NOT dependencies
    port: 8080
  periodSeconds: 10
  failureThreshold: 3
  initialDelaySeconds: 0   # startupProbe handles initial delay

Decision Matrix: Which Strategy to Use

Graceful Shutdown: The Missing Piece

Even with perfect deployment strategy, you'll see 502s during deploys if you don't handle graceful shutdown:

spec:
  terminationGracePeriodSeconds: 60
  containers:
    - name: api
      lifecycle:
        preStop:
          exec:
            # Wait for endpoints controller to remove this pod from Service
            command: ["/bin/sh", "-c", "sleep 10"]

The 10-second sleep in preStop is critical. When a pod is terminated:

Pod is marked for deletion

Endpoints controller removes it from Service (takes a few seconds)

preStop hook runs (our sleep)

SIGTERM is sent to the container

App drains in-flight requests

Pod terminates

Without that sleep, SIGTERM arrives before the endpoints update propagates, and requests still route to a terminating pod.

Final Thought

Start with rolling updates and solid health checks. That alone eliminates most downtime. Graduate to canary when your traffic volume justifies the complexity, or blue-green when you need instant rollback guarantees. The strategy matters less than getting the fundamentals right: proper probes, graceful shutdown, and the ability to roll back in seconds.

---

Frequently Asked Questions

What is zero-downtime deployment?

Zero-downtime deployment means releasing new application versions without any interruption to users. Traffic is gradually shifted from old to new instances, with health checks verifying the new version before it receives full traffic. If issues are detected, traffic automatically routes back to the old version. This requires load balancing, health checks, and database backward compatibility.

What is the difference between blue-green and canary deployments?

Blue-green deploys the new version to an identical parallel environment and switches all traffic at once — fast but all-or-nothing. Canary gradually routes a small percentage (1-10%) of traffic to the new version, increasing over time if metrics look good. Canary is safer for large user bases since problems affect fewer users, but takes longer to complete.

How do I handle database migrations with zero-downtime deployments?

Use expand-and-contract pattern: first deploy schema changes that are backward compatible (add new columns, don't rename or remove), deploy the new application code, then clean up old columns in a later release. Never deploy breaking schema changes and new code simultaneously. Tools like Flyway and Liquibase help manage migration ordering and rollback.

What is a rolling deployment and when should I use it?

Rolling deployment updates instances one at a time (or in small batches), keeping the remaining instances serving traffic. It uses fewer resources than blue-green (no parallel environment) but means both old and new versions run simultaneously during the rollout. Use rolling deployments when your application handles mixed-version traffic correctly and you want resource efficiency.

How do I implement automatic rollback?

Configure health check thresholds in your deployment tool (Kubernetes readiness probes, ECS health checks, CodeDeploy alarms) that trigger automatic rollback on failure. Monitor error rates, latency percentiles, and custom business metrics during deployment. Set a rollback window (5-15 minutes post-deploy) where elevated errors automatically revert to the previous version.

---

Related Resources

• Git Cheatsheet — 106 git commands by workflow
• DORA Metrics Calculator — DORA metrics calculator
• Production Reference Architectures — GitOps CI/CD reference architecture
• GitHub Actions CI/CD Complete Guide — Setting up deployment pipelines
• Monitoring and Alerting in Production — Monitoring deployments for rollback decisions