Blue-Green vs. Canary Deployments — Which One Won't Crash Your DB?

TL;DR Quick Fix

If your deployments keep crashing during database migrations, adopt the expand-and-contract pattern and decouple schema changes from application deploys:

# Step 1: Expand — add new column without removing old one
kubectl apply -f migration-job-expand.yaml

# Step 2: Deploy new app version that writes to BOTH columns
kubectl argo rollouts set-image my-app my-app=myrepo/app:v2.1

# Step 3: Contract — backfill data and drop old column after full rollout
kubectl apply -f migration-job-contract.yaml

Never run destructive migrations (DROP COLUMN, RENAME) during a rolling deploy. Always maintain backward compatibility for at least one release cycle.

---

Architecture Overview

Load Balancer

Blue Environment (Current)

App v1.0

Schema v1

Green Environment (New)

App v2.0

Schema v2

Shared Database (expand-contract)

Canary (5% traffic)

App v2.0

Same DB, new code path

---

Blue-Green Deployment Strategy

Blue-green gives you a full parallel environment. The switch is instant, but the cost is doubled infrastructure.

When to Use Blue-Green

• Database migrations are additive only (new tables, new columns)
• You need instant rollback capability
• You can afford running two full environments temporarily

Argo Rollouts Blue-Green Configuration

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-app
  namespace: production
spec:
  replicas: 5
  strategy:
    blueGreen:
      activeService: my-app-active
      previewService: my-app-preview
      autoPromotionEnabled: false
      prePromotionAnalysis:
        templates:
          - templateName: db-migration-check
        args:
          - name: service-name
            value: my-app-preview
      scaleDownDelaySeconds: 300
      abortScaleDownDelaySeconds: 60
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: myrepo/app:v2.0
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 5

---

Canary Deployment Strategy

Canary progressively shifts traffic. It is cheaper than blue-green but rollback is slower.

Flagger with Istio Configuration

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: my-app
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  service:
    port: 8080
    targetPort: 8080
    gateways:
      - public-gateway.istio-system.svc.cluster.local
    hosts:
      - app.example.com
  analysis:
    interval: 1m
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    metrics:
      - name: request-success-rate
        thresholdRange:
          min: 99
        interval: 1m
      - name: request-duration
        thresholdRange:
          max: 500
        interval: 1m
    webhooks:
      - name: db-schema-compatibility
        type: pre-rollout
        url: http://schema-checker.infra/validate
        timeout: 60s

Flagger with NGINX Ingress

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: my-app
spec:
  provider: nginx
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  ingressRef:
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    name: my-app
  analysis:
    interval: 1m
    threshold: 10
    maxWeight: 50
    stepWeight: 5

---

The Expand-and-Contract DB Migration Pattern

This is the golden rule for zero-downtime database changes:

Step 1: Expand Phase

# migration_v2_expand.py
from alembic import op
import sqlalchemy as sa

def upgrade():
    # Add new column (nullable to avoid breaking existing rows)
    op.add_column('users', sa.Column(
        'email_verified', sa.Boolean(),
        nullable=True, server_default='false'
    ))
    op.create_index('idx_users_email_verified', 'users', ['email_verified'])
    # DO NOT drop old columns here!

def downgrade():
    op.drop_index('idx_users_email_verified')
    op.drop_column('users', 'email_verified')

Step 2: Migrate Application Code

// user-service.ts — writes to BOTH old and new columns during transition
import { db } from './database';

export async function updateUserVerification(userId: string, verified: boolean) {
  await db.query(

    UPDATE users
    SET
      email_verified = $1,       -- new column (expand phase)
      is_verified = $1           -- old column (backward compat)
    WHERE id = $2
  , [verified, userId]);
}

Step 3: Contract Phase (After Full Rollout)

# migration_v3_contract.py — run ONLY after all instances use v2+
from alembic import op
import sqlalchemy as sa

def upgrade():
    op.drop_column('users', 'is_verified')

def downgrade():
    op.add_column('users', sa.Column(
        'is_verified', sa.Boolean(), server_default='false'
    ))

---

Database Backward Compatibility Rules

#!/bin/bash
# pre-deploy-db-check.sh — validate migration safety
set -euo pipefail

MIGRATION_FILE=$1
echo "Checking migration for destructive operations..."

FORBIDDEN_OPS=("DROP COLUMN" "DROP TABLE" "RENAME COLUMN" "ALTER COLUMN.*TYPE" "DROP INDEX")

for op in "${FORBIDDEN_OPS[@]}"; do
  if grep -qiE "$op" "$MIGRATION_FILE"; then
    echo "BLOCKED: Found destructive operation '$op' in $MIGRATION_FILE"
    echo "Use expand-and-contract pattern instead."
    exit 1
  fi
done

echo "Migration is safe for rolling deployment"

---

Monitoring Deployment Health

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: db-migration-check
spec:
  args:
    - name: service-name
  metrics:
    - name: error-rate
      interval: 30s
      successCondition: result[0] < 0.01
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}", code=~"5.."}[2m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[2m]))
    - name: db-connection-errors
      interval: 30s
      successCondition: result[0] == 0
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(increase(db_connection_errors_total{service="{{args.service-name}}"}[2m]))

---

Comparison Table

---

FAQ

Can I combine blue-green and canary strategies?

Yes. A common pattern is blue-green at the infrastructure layer (separate clusters or namespaces) with canary at the traffic layer. Argo Rollouts supports hybrid strategies where you deploy to a preview environment first, validate, then do a canary rollout to production traffic.

What if my migration takes longer than the deployment timeout?

Decouple migrations from deployments entirely. Run migrations as separate Kubernetes Jobs that execute before the Rollout begins. Use init containers or pre-sync hooks in ArgoCD to ensure migrations complete before new pods start.

How do I handle foreign key constraints during expand-and-contract?

Add foreign keys as NOT VALID first, then validate them in a separate transaction. This avoids locking the referenced table during the constraint check.

Should I use a separate database for blue-green?

Only if your schema changes are truly incompatible. Most teams share a single database with expand-and-contract migrations. A separate database requires data synchronization, which introduces significant complexity and potential data loss.

How do I test migrations before they hit production?

Run migrations against a production-clone database in CI. Use pg_dump with schema-only to create a schema snapshot, and validate migrations against it. Also use shadow databases in staging that mirror production schema.

---

Related Resources

• GitOps with ArgoCD Complete Guide
• Kubernetes Rolling Updates and Rollback Strategies
• CI/CD Pipeline Optimization Guide
• Feature Flags for Safe Deployments
• Database Observability and Performance