Skip to main content
Infrastructure as Code·5 min read

Fixing Terraform Drift Before It Causes a Production Incident

Detect and prevent Terraform drift caused by manual console changes, automation scripts, and SDK calls. Set up automated drift alerts with driftctl, Atlantis, and SCPs.

DT

DevOps Engineer & Technical Writer

TL;DR — Quick Fix

Detect drift right now with a single command:

# Option 1: Native Terraform drift detection

terraform plan -detailed-exitcode

# Exit code 2 = drift detected

# Option 2: driftctl for comprehensive drift analysis

driftctl scan --from tfstate://terraform.tfstate

# Shows managed, unmanaged, and changed resources

# Option 3: Quick drift check across all workspaces

for dir in envs/*/; do

echo "=== Checking $dir ==="

(cd "$dir" && terraform plan -detailed-exitcode -compact-warnings 2>&1 | tail -5)

done

---

What Causes Terraform Drift

Drift occurs when real infrastructure diverges from what Terraform expects.

Terraform Drift — How It Happens

Terraform State (expected)

Real Infrastructure (actual)

DRIFT GAP

Console Changes

SG rules, tags added

via AWS Console

SDK/CLI Scripts

Automation bypassing

IaC workflows

Auto-Scaling

ASG changes count

outside Terraform

Hotfixes

Emergency changes

never codified

Prevention Strategy

SCPs block console writes + Atlantis for all applies

Nightly drift scans + lifecycle ignore_changes for dynamic attrs

---

Detecting Drift with driftctl

# Install driftctl

brew install driftctl

# Full scan against current state

driftctl scan --from tfstate://terraform.tfstate

# Scan with S3 remote backend

driftctl scan --from tfstate+s3://my-bucket/env/production/terraform.tfstate

# Generate coverage report

driftctl scan --from tfstate://terraform.tfstate --output html://drift-report.html

# Scan only specific resource types

driftctl scan --from tfstate://terraform.tfstate \

--filter "Type=='aws_security_group_rule' || Type=='aws_iam_policy'"

---

Preventing Drift with SCPs

{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "DenyConsoleWriteActions",

"Effect": "Deny",

"Action": [

"ec2:AuthorizeSecurityGroupIngress",

"ec2:RevokeSecurityGroupIngress",

"ec2:ModifyInstanceAttribute",

"rds:ModifyDBInstance",

"s3:PutBucketPolicy"

],

"Resource": "*",

"Condition": {

"StringNotLike": {

"aws:PrincipalArn": [

"arn:aws:iam::*:role/AtlantisRole",

"arn:aws:iam::*:role/TerraformCIRole"

]

}

}

}

]

}

---

Automated Drift Alerts with CI/CD

# .github/workflows/drift-detection.yml

name: Terraform Drift Detection

on:

schedule:

- cron: '0 /6 '

workflow_dispatch: {}

jobs:

detect-drift:

runs-on: ubuntu-latest

strategy:

matrix:

environment: [staging, production]

steps:

- uses: actions/checkout@v4

- uses: hashicorp/setup-terraform@v3

with:

terraform_version: 1.7.0

- name: Terraform Init

run: terraform init

working-directory: envs/${{ matrix.environment }}

- name: Check for Drift

id: drift

run: |

terraform plan -detailed-exitcode -compact-warnings 2>&1 | tee plan.txt

echo "exitcode=$?" >> $GITHUB_OUTPUT

working-directory: envs/${{ matrix.environment }}

continue-on-error: true

- name: Alert on Drift

if: steps.drift.outputs.exitcode == '2'

run: |

curl -X POST "${{ secrets.SLACK_WEBHOOK }}" \

-H 'Content-Type: application/json' \

-d "{\"text\": \"Drift detected in ${{ matrix.environment }}\"}"

---

Atlantis for Drift Prevention

# atlantis.yaml — Enforce all changes through PR workflow

version: 3

projects:

- name: production

dir: envs/production

workflow: production-strict

apply_requirements:

- approved

- mergeable

autoplan:

when_modified: ["/.tf", "../modules//.tf"]

enabled: true

workflows:

production-strict:

plan:

steps:

- init

- plan

- run: tflint --module

- run: conftest test $PLANFILE --policy policies/

apply:

steps:

- apply

---

Remediation Workflows

# Handle expected drift with lifecycle blocks

resource "aws_autoscaling_group" "app" {

name = "app-asg"

min_size = 3

max_size = 20

desired_capacity = 3

lifecycle {

ignore_changes = [desired_capacity]

}

}

resource "aws_security_group" "app" {

name = "app-sg"

description = "Application security group"

vpc_id = var.vpc_id

lifecycle {

ignore_changes = [ingress, egress]

}

}

#!/bin/bash

# Remediation: import unmanaged resources into state

set -euo pipefail

DRIFT_REPORT=$(driftctl scan --from tfstate://terraform.tfstate --output json://-)

UNMANAGED=$(echo "$DRIFT_REPORT" | jq -r '.unmanaged[] | "\(.type) \(.id)"')

echo "Unmanaged resources found:"

echo "$UNMANAGED"

echo ""

echo "To import into Terraform state:"

echo "$UNMANAGED" | while read -r type id; do

echo " terraform import ${type}.imported_${id//[-.]/_} ${id}"

done

---

FAQ

Q: How is driftctl different from terraform plan?

A: terraform plan only shows drift on resources already in state. driftctl also finds unmanaged resources — things in AWS not managed by any Terraform state.

Q: Won't SCPs break incident response?

A: Use conditional SCPs that exempt break-glass roles. Create a dedicated incident role assumable only through approval workflows.

Q: How do I handle drift from AWS-managed services?

A: Use lifecycle { ignore_changes } for auto-modified attributes (ASG desired_capacity, ECS task count). Document each ignore.

Q: Should I auto-remediate drift or just alert?

A: Start with alerting. Auto-remediation is risky since it might revert intentional hotfixes. Use it only for well-understood, non-critical patterns.

Q: How often does drift happen in practice?

A: Without prevention, 15-30% of resources drift within 30 days. With SCPs and Atlantis, this drops to under 2%.

---