Terraform State Lock Deadlocks — Recovery, Prevention & Production Fixes

TL;DR — Quick Fix

# 1. Check who holds the lock
terraform force-unlock <LOCK_ID>

# 2. If using DynamoDB (AWS), check the lock table
aws dynamodb scan --table-name terraform-locks --output table

# 3. If state is corrupted, pull the last known good version
aws s3api list-object-versions --bucket my-tf-state --prefix env/prod/terraform.tfstate

Warning: Never run force-unlock without understanding why the lock exists. A running terraform apply in another pipeline holds a legitimate lock.

---

What Causes State Lock Deadlocks?

Terraform uses a locking mechanism to prevent concurrent writes to the same state file. When a terraform plan or terraform apply starts, it acquires a lock. When it finishes, it releases the lock.

The deadlock happens when the process holding the lock terminates without releasing it — leaving a stale lock that blocks all future operations.

Step 1: Identify the Stale Lock

When you see this error:

Error: Error acquiring the state lock

Error message: ConditionalCheckFailedException: The conditional request failed
Lock Info:
  ID:        a1b2c3d4-e5f6-7890-abcd-ef1234567890
  Path:      s3://my-tf-state/env/prod/terraform.tfstate
  Operation: OperationTypeApply
  Who:       runner@github-actions-12345
  Version:   1.9.3
  Created:   2026-08-01 10:15:32.123456 +0000 UTC

Before force-unlocking, verify:

Is there actually a running apply somewhere?

When was the lock created? (If it's hours/days old, it's definitely stale)

Who created it? (Check CI/CD pipeline history)

# Check DynamoDB for the lock entry (AWS S3 backend)
aws dynamodb get-item \
  --table-name terraform-locks \
  --key '{"LockID": {"S": "my-tf-state/env/prod/terraform.tfstate-md5"}}' \
  --output json

Step 2: Safely Force-Unlock

Once you've confirmed the lock is stale:

# Use the Lock ID from the error message
terraform force-unlock a1b2c3d4-e5f6-7890-abcd-ef1234567890

If Terraform CLI isn't available or the state is deeply corrupted, manually delete the DynamoDB item:

# Nuclear option — only if terraform force-unlock doesn't work
aws dynamodb delete-item \
  --table-name terraform-locks \
  --key '{"LockID": {"S": "my-tf-state/env/prod/terraform.tfstate-md5"}}'

Step 3: Verify State Integrity After Unlock

After unlocking, check that the state file isn't corrupted:

# Pull the current state
terraform state pull > current-state.json

# Validate it's valid JSON
python3 -c "import json; json.load(open('current-state.json'))"

# Check resource count matches expectations
terraform state list | wc -l

# Run plan to see if drift exists
terraform plan

Recovering Corrupted State Files

If the crash happened mid-write, the state file might be incomplete or corrupted.

From S3 Versioning (Best Case)

# List previous versions of the state file
aws s3api list-object-versions \
  --bucket my-tf-state \
  --prefix env/prod/terraform.tfstate \
  --max-items 5

# Download a previous good version
aws s3api get-object \
  --bucket my-tf-state \
  --key env/prod/terraform.tfstate \
  --version-id "abc123previousversion" \
  restored-state.json

# Verify and push the restored state
terraform state push restored-state.json

From Terraform Cloud/Enterprise

# List state versions in Terraform Cloud UI
# Settings > States > select version > "Rollback to this state"

Last Resort: Rebuild State

If no backup exists:

# Import resources one by one
terraform import aws_instance.web i-1234567890abcdef0
terraform import aws_s3_bucket.data my-bucket-name
terraform import aws_rds_instance.db my-database-identifier

# Or use terraformer to auto-discover and import
terraformer import aws --resources=ec2,s3,rds --regions=us-east-1

Prevention: Backend Configuration Best Practices

S3 + DynamoDB Backend (AWS)

terraform {
  backend "s3" {
    bucket         = "mycompany-terraform-state"
    key            = "env/prod/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"
  }
}

Ensure your S3 bucket has versioning enabled:

resource "aws_s3_bucket_versioning" "state" {
  bucket = aws_s3_bucket.terraform_state.id
  versioning_configuration {
    status = "Enabled"
  }
}

GCS Backend (Google Cloud)

terraform {
  backend "gcs" {
    bucket = "mycompany-terraform-state"
    prefix = "env/prod"
    # GCS backend has built-in locking via object generation
  }
}

Azure Blob Backend

terraform {
  backend "azurerm" {
    resource_group_name  = "terraform-state-rg"
    storage_account_name = "tfstate12345"
    container_name       = "tfstate"
    key                  = "prod.terraform.tfstate"
    # Azure uses blob leases for locking
  }
}

CI/CD Pipeline Protection Patterns

Pattern 1: Timeout-Aware Locking

# GitHub Actions — ensure cleanup on timeout
jobs:
  terraform:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3

      - name: Terraform Apply
        id: apply
        run: terraform apply -auto-approve
        timeout-minutes: 20

      # If the job is cancelled, try to release the lock
      - name: Force Unlock on Failure
        if: cancelled()
        run: |
          LOCK_ID=$(terraform force-unlock -force 2>&1 | grep -oP '[a-f0-9-]{36}' || true)
          if [ -n "$LOCK_ID" ]; then
            terraform force-unlock -force "$LOCK_ID"
          fi

Pattern 2: Prevent Concurrent Runs

# GitHub Actions — concurrency group prevents parallel applies
concurrency:
  group: terraform-prod
  cancel-in-progress: false  # Don't cancel running applies!

Pattern 3: Atlantis / Spacelift Lock Management

If you're using Atlantis for Terraform automation, it handles locking at the project level:

# atlantis.yaml
version: 3
projects:
  - dir: environments/prod
    workspace: default
    autoplan:
      enabled: true
      when_modified: [".tf", ".tfvars"]
    apply_requirements: [approved, mergeable]

Monitoring & Alerting for Stale Locks

#!/bin/bash
# Script: check-stale-locks.sh
# Run via cron every 15 minutes

TABLE="terraform-locks"
MAX_AGE_MINUTES=30

LOCKS=$(aws dynamodb scan --table-name "$TABLE" --output json)
LOCK_COUNT=$(echo "$LOCKS" | jq '.Count')

if [ "$LOCK_COUNT" -gt 0 ]; then
  echo "$LOCKS" | jq -r '.Items[] | .Info.S' | while read -r info; do
    CREATED=$(echo "$info" | jq -r '.Created')
    AGE_MINUTES=$(( ($(date +%s) - $(date -d "$CREATED" +%s)) / 60 ))

    if [ "$AGE_MINUTES" -gt "$MAX_AGE_MINUTES" ]; then
      echo "ALERT: Stale lock detected! Age: ${AGE_MINUTES}m, Info: $info"
      # Send to Slack/PagerDuty
    fi
  done
fi

State File Structure — What Goes Wrong During a Crash

Understanding the state file format helps when debugging corruption:

{
  "version": 4,
  "terraform_version": "1.9.3",
  "serial": 142,
  "lineage": "a1b2c3d4-uuid-here",
  "outputs": {},
  "resources": []
}

What happens during a crash:

• serial might increment without resources updating (partial write)
• Resources might be created in the cloud but not recorded in state (orphaned resources)
• Resources might be recorded as destroyed but still exist (state says deleted, cloud says alive)

After recovery, always run terraform plan to detect drift between state and real infrastructure.

---

Frequently Asked Questions

Is it safe to run terraform force-unlock?

It's safe only if you've confirmed no other Terraform process is actively running against that state. If another apply is in progress, force-unlocking can lead to state corruption from concurrent writes. Always check CI/CD pipeline history and verify the lock creation timestamp before unlocking.

How do I prevent state lock deadlocks in CI/CD?

Use concurrency groups to prevent parallel Terraform runs against the same state file. Set appropriate timeouts on CI/CD jobs. Add cleanup steps that run on cancellation or failure. Consider tools like Atlantis or Spacelift that manage locking at a higher level with automatic recovery.

What happens if my Terraform state file gets corrupted?

If you have S3 versioning enabled (or equivalent), restore a previous version. Run terraform plan after restoration to detect any drift between the restored state and actual infrastructure. Resources created after the backup will appear as "to be created" in the plan — handle them with terraform import.

How do I split a large Terraform state to reduce lock contention?

Use separate state files per environment and per service layer. Structure your code into modules with independent backends. For example: networking/, compute/, database/ each with their own state file. This way, a lock on the database state doesn't block networking changes.

Can two people run terraform plan at the same time?

terraform plan with remote backends also acquires a lock (to read consistent state). However, plans are quick and the lock duration is short. If you need concurrent reads, consider using -lock=false for plan-only operations in development (never for apply).

---

Related Resources

• Terraform State Management in Production — Backend configuration and state organization
• Terraform Modules Best Practices — Structuring code to reduce blast radius
• Terraform Import Existing Resources — Bring existing infra under Terraform management