TL;DR — Quick Fix
If your staging environment doesn't match production, run this drift detection immediately:
# Install driftctl and scan for drift between state and reality
brew install driftctl
driftctl scan --from tfstate://terraform.tfstate --output json://drift-report.json
# Quick comparison of Terraform outputs between environments
diff <(cd envs/staging && terraform output -json) \
<(cd envs/production && terraform output -json) | jq .
This catches the most common drift sources — manual console changes, automation scripts bypassing IaC, and forgotten hotfixes.
---
Why Environments Drift Apart
Environment drift is the silent killer of deployment confidence. It happens when staging and production diverge in ways that aren't captured in code.
---
Immutable Environment Templates with Terraform
The key principle: environments should be instantiated from the same module with different variable files.
# modules/environment/main.tf
module "vpc" {
source = "../modules/vpc"
cidr_block = var.vpc_cidr
environment = var.environment
availability_zones = var.azs
}
module "eks" {
source = "../modules/eks"
cluster_name = "${var.project}-${var.environment}"
cluster_version = var.k8s_version # Same version across envs
node_groups = var.node_groups
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnet_ids
}
module "rds" {
source = "../modules/rds"
engine_version = var.rds_engine_version # Pinned, same everywhere
instance_class = var.rds_instance_class
multi_az = var.environment == "production" ? true : false
deletion_protection = var.environment == "production" ? true : false
}
# envs/staging/terraform.tfvars
environment = "staging"
vpc_cidr = "10.1.0.0/16"
k8s_version = "1.29"
rds_engine_version = "15.4"
rds_instance_class = "db.t3.medium"
node_groups = {
default = { min = 2, max = 5, instance_type = "t3.large" }
}
# envs/production/terraform.tfvars
environment = "production"
vpc_cidr = "10.0.0.0/16"
k8s_version = "1.29"
rds_engine_version = "15.4"
rds_instance_class = "db.r5.xlarge"
node_groups = {
default = { min = 3, max = 20, instance_type = "m5.xlarge" }
}
---
Kustomize Overlays for Environment Parity
# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
- configmap.yaml
- hpa.yaml
# overlays/staging/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patchesStrategicMerge:
- replica-patch.yaml
configMapGenerator:
- name: app-config
behavior: merge
literals:
- LOG_LEVEL=debug
- FEATURE_NEW_UI=true
# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patchesStrategicMerge:
- replica-patch.yaml
- resource-patch.yaml
configMapGenerator:
- name: app-config
behavior: merge
literals:
- LOG_LEVEL=warn
- FEATURE_NEW_UI=false
---
Automated Drift Detection with driftctl
#!/bin/bash
# scripts/detect-drift.sh — Run nightly via cron or CI
set -euo pipefail
ENVS=("staging" "production")
SLACK_WEBHOOK="${SLACK_DRIFT_WEBHOOK}"
for env in "${ENVS[@]}"; do
echo "Scanning ${env} for drift..."
cd "envs/${env}"
DRIFT_OUTPUT=$(driftctl scan \
--from tfstate://terraform.tfstate \
--output json://- 2>/dev/null)
UNMANAGED=$(echo "$DRIFT_OUTPUT" | jq '.summary.total_unmanaged')
CHANGED=$(echo "$DRIFT_OUTPUT" | jq '.summary.total_changed')
if [[ "$UNMANAGED" -gt 0 ]] || [[ "$CHANGED" -gt 0 ]]; then
curl -s -X POST "$SLACK_WEBHOOK" \
-H 'Content-Type: application/json' \
-d "{
\"text\": \"Warning: Drift detected in ${env}\nUnmanaged: ${UNMANAGED}\nChanged: ${CHANGED}\"
}"
fi
cd ../..
done
---
Policy Enforcement with Atlantis and OPA
# atlantis.yaml
version: 3
projects:
- name: staging
dir: envs/staging
workflow: standard
autoplan:
when_modified: ["/.tf", "../modules//.tf"]
- name: production
dir: envs/production
workflow: production
autoplan:
when_modified: ["/.tf", "../modules//.tf"]
workflows:
production:
plan:
steps:
- init
- plan
- run: conftest test $PLANFILE --policy ../policies/
apply:
steps:
- run: echo "Requires 2 approvals for production"
- apply
# scripts/env_compare.py — Compare environment configurations
import subprocess
import json
import sys
def get_terraform_output(env_dir):
result = subprocess.run(
["terraform", "output", "-json"],
cwd=env_dir, capture_output=True, text=True
)
return json.loads(result.stdout)
def compare_environments(staging_dir, prod_dir):
staging = get_terraform_output(staging_dir)
prod = get_terraform_output(prod_dir)
differences = []
critical_keys = ["k8s_version", "rds_engine_version", "vpc_flow_logs_enabled"]
for key in critical_keys:
stg_val = staging.get(key, {}).get("value")
prd_val = prod.get(key, {}).get("value")
if stg_val != prd_val:
differences.append({
"key": key, "staging": stg_val,
"production": prd_val, "severity": "CRITICAL"
})
return differences
if __name__ == "__main__":
diffs = compare_environments("envs/staging", "envs/production")
if diffs:
for d in diffs:
print(f" {d['key']}: staging={d['staging']} vs prod={d['production']}")
sys.exit(1)
print("Critical configurations match across environments")
---
FAQ
Q: How often should I run drift detection?
A: Run it at minimum once daily (nightly cron). For critical infrastructure, run every 4-6 hours. Integrate with your CI pipeline so every PR triggers a drift check before apply.
Q: What about intentional differences between environments?
A: Intentional differences should be captured in your tfvars or overlay files. Drift detection catches unintentional changes. Use driftctl's .driftignore to whitelist known differences.
Q: Should staging be an exact replica of production?
A: Not necessarily. Staging should match production in architecture, software versions, and configuration shape. It can differ in scale. The key is that code deployed to staging exercises the same paths it will in production.
Q: How do I handle emergency hotfixes that bypass IaC?
A: Create a hotfix reconciliation process. After any manual change, require a follow-up PR within 24 hours that codifies the change. Use AWS Config rules or SCPs to alert on out-of-band changes.
Q: Can I prevent console access entirely?
A: You can restrict with AWS SCPs that deny write actions from the console. A less extreme approach is allowing read-only console access and requiring all writes through CI/CD pipelines with Atlantis.
---