TL;DR Quick Fix
Protect your critical infrastructure from accidental deletion right now:
# Add to every critical resource in Terraform
resource "aws_db_instance" "production" {
# ... configuration ...
deletion_protection = true
lifecycle {
prevent_destroy = true
}
}
resource "aws_s3_bucket" "data_lake" {
bucket = "company-data-lake-prod"
lifecycle {
prevent_destroy = true
}
}
Then add an SCP to prevent deletions at the AWS Organization level:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PreventRDSDeletion",
"Effect": "Deny",
"Action": [
"rds:DeleteDBInstance",
"rds:DeleteDBCluster"
],
"Resource": "*",
"Condition": {
"StringNotEquals": {
"aws:PrincipalTag/BreakGlass": "true"
}
}
}
]
}
---
Architecture Overview
---
Accidental Destroy Scenarios
Real-world incidents that cost teams data and downtime:
# Scenario 1: Wrong workspace selected
terraform workspace select production # thought it was staging
terraform destroy -auto-approve # goodbye production
# Scenario 2: State corruption causes "drift" detection
# Terraform decides resources need to be replaced, not updated
# Plan shows: 1 to destroy, 1 to create (but the destroy happens first)
# Scenario 3: Module refactoring causes resource recreation
# Moving a resource to a different module path triggers destroy+create
# terraform state mv can prevent this, but only if you catch it first
---
Terraform prevent_destroy Lifecycle Rule
# modules/database/main.tf
resource "aws_db_instance" "main" {
identifier = "${var.env}-${var.app}-db"
engine = "postgres"
engine_version = "15.4"
instance_class = var.instance_class
# AWS-level deletion protection
deletion_protection = true
# Enable automated backups
backup_retention_period = 30
backup_window = "03:00-04:00"
# Terraform-level deletion protection
lifecycle {
prevent_destroy = true
ignore_changes = [engine_version] # handled by maintenance window
}
tags = {
Environment = var.env
ManagedBy = "terraform"
Critical = "true"
}
}
resource "aws_s3_bucket" "critical_data" {
bucket = "${var.org}-${var.env}-critical-data"
lifecycle {
prevent_destroy = true
}
}
resource "aws_s3_bucket_versioning" "critical_data" {
bucket = aws_s3_bucket.critical_data.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_dynamodb_table" "app_state" {
name = "${var.env}-app-state"
billing_mode = "PAY_PER_REQUEST"
hash_key = "id"
deletion_protection_enabled = true
point_in_time_recovery {
enabled = true
}
lifecycle {
prevent_destroy = true
}
attribute {
name = "id"
type = "S"
}
}
---
AWS Deletion Protection Settings
# Enable deletion protection across critical services
# RDS
resource "aws_db_instance" "prod" {
deletion_protection = true
skip_final_snapshot = false
final_snapshot_identifier = "${var.app}-final-${formatdate("YYYYMMDD", timestamp())}"
}
# EKS
resource "aws_eks_cluster" "prod" {
name = "production"
# No native deletion protection, use SCP instead
}
# ElastiCache
resource "aws_elasticache_replication_group" "prod" {
automatic_failover_enabled = true
# Create final snapshot before any deletion
final_snapshot_identifier = "redis-final"
}
# S3 - prevent bucket deletion if objects exist
resource "aws_s3_bucket" "important" {
bucket = "important-data"
force_destroy = false # THIS IS THE DEFAULT - never set to true in prod
}
# Load Balancer
resource "aws_lb" "prod" {
enable_deletion_protection = true
}
---
Service Control Policies (SCPs)
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PreventCriticalDeletions",
"Effect": "Deny",
"Action": [
"rds:DeleteDBInstance",
"rds:DeleteDBCluster",
"dynamodb:DeleteTable",
"s3:DeleteBucket",
"eks:DeleteCluster",
"elasticloadbalancing:DeleteLoadBalancer"
],
"Resource": "*",
"Condition": {
"StringNotEquals": {
"aws:PrincipalTag/BreakGlass": "true"
}
}
},
{
"Sid": "PreventS3BucketDeletionWithObjects",
"Effect": "Deny",
"Action": [
"s3:DeleteBucket"
],
"Resource": "arn:aws:s3:::-prod-"
},
{
"Sid": "PreventDisablingBackups",
"Effect": "Deny",
"Action": [
"rds:ModifyDBInstance"
],
"Resource": "*",
"Condition": {
"NumericEquals": {
"rds:BackupRetentionPeriod": "0"
}
}
}
]
}
---
Terraform State Protection
# backend.tf — secure state configuration
terraform {
backend "s3" {
bucket = "company-terraform-state"
key = "production/infrastructure.tfstate"
region = "us-east-1"
encrypt = true
dynamodb_table = "terraform-state-lock"
# Enable versioning on the state bucket
# (configured separately on the bucket itself)
}
}
# Protect the state bucket itself
aws s3api put-bucket-versioning \
--bucket company-terraform-state \
--versioning-configuration Status=Enabled
# Add MFA delete requirement for state bucket
aws s3api put-bucket-versioning \
--bucket company-terraform-state \
--versioning-configuration Status=Enabled,MFADelete=Enabled \
--mfa "arn:aws:iam::123456789012:mfa/admin 123456"
# Block public access
aws s3api put-public-access-block \
--bucket company-terraform-state \
--public-access-block-configuration \
BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
---
Automated Backup Verification
#!/bin/bash
# verify-backups.sh — ensure backups are valid and restorable
set -euo pipefail
echo "=== Backup Verification Report ==="
echo "Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
# Check RDS automated backups
echo ""
echo "--- RDS Backup Status ---"
aws rds describe-db-instances \
--query 'DBInstances[].{ID:DBInstanceIdentifier,Backup:BackupRetentionPeriod,LastBackup:LatestRestorableTime}' \
--output table
# Verify S3 versioning is enabled on critical buckets
echo ""
echo "--- S3 Versioning Status ---"
for bucket in $(aws s3api list-buckets --query 'Buckets[?contains(Name, prod)].Name' --output text); do
STATUS=$(aws s3api get-bucket-versioning --bucket "$bucket" --query 'Status' --output text)
echo " $bucket: $STATUS"
if [ "$STATUS" != "Enabled" ]; then
echo " WARNING: Versioning not enabled on $bucket"
fi
done
# Check DynamoDB PITR status
echo ""
echo "--- DynamoDB Point-in-Time Recovery ---"
for table in $(aws dynamodb list-tables --query 'TableNames[?contains(@, prod)]' --output text); do
PITR=$(aws dynamodb describe-continuous-backups \
--table-name "$table" \
--query 'ContinuousBackupsDescription.PointInTimeRecoveryDescription.PointInTimeRecoveryStatus' \
--output text)
echo " $table: $PITR"
done
---
CI/CD Safety Gates for Terraform
# .github/workflows/terraform-plan.yml
name: Terraform Plan Review
on:
pull_request:
paths: ['terraform/**']
jobs:
plan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Terraform Plan
id: plan
run: terraform plan -out=tfplan -detailed-exitcode
- name: Check for destructive changes
run: |
# Parse the plan for any destroy actions
terraform show -json tfplan | \
jq -r '.resource_changes[] | select(.change.actions[] == "delete") | .address' > destroys.txt
if [ -s destroys.txt ]; then
echo "DESTRUCTIVE CHANGES DETECTED:"
cat destroys.txt
echo ""
echo "This PR requires manual approval from the infrastructure team."
gh pr comment ${{ github.event.number }} \
--body "## Destructive Changes Detected
The following resources will be DESTROYED:
$(cat destroys.txt | sed 's/^/- /')
Requires approval from @infra-team"
exit 1
fi
---
Break-Glass Procedure for Intentional Deletion
#!/bin/bash
# break-glass-delete.sh — controlled deletion with audit trail
set -euo pipefail
RESOURCE=$1
REASON=$2
APPROVER=$3
echo "=== BREAK-GLASS DELETION REQUEST ==="
echo "Resource: $RESOURCE"
echo "Reason: $REASON"
echo "Approver: $APPROVER"
echo "Operator: $(aws sts get-caller-identity --query 'Arn' --output text)"
echo "Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
# Log to audit trail
aws cloudwatch put-metric-data \
--namespace "Infrastructure/BreakGlass" \
--metric-name "DeletionRequest" \
--value 1 \
--dimensions Resource=$RESOURCE
# Verify approval exists
echo ""
echo "Please confirm deletion by typing the resource name:"
read CONFIRMATION
if [ "$CONFIRMATION" != "$RESOURCE" ]; then
echo "Confirmation failed. Aborting."
exit 1
fi
echo "Proceeding with break-glass deletion..."
---
FAQ
What if I need to replace a resource that has prevent_destroy?
Temporarily remove the lifecycle block, run terraform apply to delete the old resource, then add the lifecycle block back on the new resource. Better yet, use terraform state mv to rename the resource in state without destroying it. Always do this in a reviewed PR, never ad-hoc.
Does prevent_destroy protect against terraform state rm?
No. terraform state rm removes the resource from state tracking without destroying it in AWS. The resource continues to exist but Terraform no longer manages it. This is actually useful for migrating resources between state files but should be audited carefully.
How do I handle legitimate infrastructure decommissioning?
Create a documented decommissioning process: verify backups exist, take a final snapshot, get written approval, remove prevent_destroy in a PR, apply the deletion in a separate PR, and verify the final snapshot is accessible. Log everything.
Should I use prevent_destroy on every resource?
No. Use it on stateful resources (databases, storage, clusters) and resources that would cause significant downtime if recreated (load balancers, DNS records). Stateless resources (Lambda functions, IAM roles) can be safely recreated without prevent_destroy.
What about Terraform Cloud / Enterprise run protections?
Terraform Cloud offers Sentinel policies that can block destroys at the plan level. This adds another layer: even if prevent_destroy is removed from code, Sentinel can still block the destroy if the resource matches a protection rule. Use both for defense in depth.
---