Secrets Management in DevOps: Stop Hardcoding, Start Encrypting

The Cost of a Leaked Secret

In 2023, a developer committed an AWS access key to a public GitHub repo. Within 4 minutes — not hours, minutes — automated scanners found it. Within 10 minutes, the attacker had spun up $50,000 worth of EC2 instances for crypto mining. GitHub's secret scanning caught it and notified the developer, but by then the damage was done.

Hardcoded secrets are the most preventable security incident in DevOps. Yet I still see them in production codebases every week: database passwords in docker-compose files, API keys in environment variables committed to git, TLS certificates in config maps.

Here's how to do it right.

The Golden Rules

Never commit secrets to version control. Not even in private repos. Not even "temporarily." Git history is forever.

Secrets should be injected at runtime. The application pulls secrets when it starts, not at build time.

Rotate secrets automatically. If rotation is painful, you'll never do it.

Audit access. Know who accessed which secret, and when.

Encrypt at rest and in transit. Always.

Option 1: AWS Secrets Manager

Best for: Teams already in AWS. Native integration with Lambda, ECS, RDS, and EKS.

Storing a Secret

aws secretsmanager create-secret \
  --name "production/api/database" \
  --description "Production PostgreSQL credentials" \
  --secret-string '{"username":"app_user","password":"s3cur3P@ss!","host":"db.internal","port":"5432","dbname":"orders"}'

Automatic Rotation With Lambda

aws secretsmanager rotate-secret \
  --secret-id "production/api/database" \
  --rotation-lambda-arn "arn:aws:lambda:us-east-1:123456789012:function:rotate-db-secret" \
  --rotation-rules '{"ScheduleExpression":"rate(30 days)"}'

Accessing in ECS Task Definition

{
  "containerDefinitions": [
    {
      "name": "api",
      "image": "myregistry/api:v2.3.1",
      "secrets": [
        {
          "name": "DB_PASSWORD",
          "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:production/api/database:password::"
        },
        {
          "name": "DB_USERNAME",
          "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:production/api/database:username::"
        }
      ]
    }
  ],
  "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole"
}

Terraform Integration

resource "aws_secretsmanager_secret" "db_credentials" {
  name                    = "${var.environment}/api/database"
  recovery_window_in_days = 7
  kms_key_id              = aws_kms_key.secrets.arn

  tags = {
    Environment = var.environment
    Service     = "api"
    ManagedBy   = "terraform"
  }
}

resource "aws_secretsmanager_secret_version" "db_credentials" {
  secret_id = aws_secretsmanager_secret.db_credentials.id
  secret_string = jsonencode({
    username = "app_user"
    password = random_password.db.result
    host     = aws_db_instance.main.address
    port     = "5432"
    dbname   = "orders"
  })
}

# Reference in ECS task
resource "aws_ecs_task_definition" "api" {
  # ...
  container_definitions = jsonencode([
    {
      name  = "api"
      image = "${aws_ecr_repository.api.repository_url}:${var.image_tag}"
      secrets = [
        {
          name      = "DATABASE_URL"
          valueFrom = aws_secretsmanager_secret.db_credentials.arn
        }
      ]
    }
  ])
}

Option 2: HashiCorp Vault

Best for: Multi-cloud, on-prem, or complex environments needing dynamic secrets, leases, and fine-grained access control.

Setting Up Vault Secrets

# Enable KV secrets engine
vault secrets enable -path=secret kv-v2

# Store a secret
vault kv put secret/production/api \
  db_password="s3cur3P@ss!" \
  api_key="sk-prod-abc123" \
  redis_url="redis://redis.internal:6379"

# Read it back
vault kv get -field=db_password secret/production/api

Dynamic Database Credentials

This is Vault's killer feature. Instead of static passwords, Vault generates short-lived credentials on demand:

# Configure database secrets engine
vault secrets enable database

vault write database/config/production-postgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="api-readonly","api-readwrite" \
  connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/orders" \
  username="vault_admin" \
  password="admin_pass"

# Define a role with a TTL
vault write database/roles/api-readonly \
  db_name=production-postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Application requests temporary credentials
vault read database/creds/api-readonly
# Returns: username=v-app-readonly-abc123, password=temp-pass-xyz, lease_duration=1h

Kubernetes Integration With Vault Agent Injector

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "api-service"
        vault.hashicorp.com/agent-inject-secret-db: "secret/data/production/api"
        vault.hashicorp.com/agent-inject-template-db: |
          {{- with secret "secret/data/production/api" -}}
          export DB_PASSWORD="{{ .Data.data.db_password }}"
          export API_KEY="{{ .Data.data.api_key }}"
          {{- end }}
    spec:
      serviceAccountName: api-service
      containers:
        - name: api
          image: myregistry/api:v2.3.1
          command: ["/bin/sh", "-c", "source /vault/secrets/db && ./start-app"]

Vault Policy (Least Privilege)

# api-service-policy.hcl
path "secret/data/production/api" {
  capabilities = ["read"]
}

path "database/creds/api-readonly" {
  capabilities = ["read"]
}

# Deny everything else
path "secret/*" {
  capabilities = ["deny"]
}

vault policy write api-service api-service-policy.hcl
vault write auth/kubernetes/role/api-service \
  bound_service_account_names=api-service \
  bound_service_account_namespaces=production \
  policies=api-service \
  ttl=1h

Option 3: SOPS (Secrets OPerationS)

Best for: GitOps workflows where you want encrypted secrets stored alongside code. Great with Flux or ArgoCD.

Encrypting With AWS KMS

# Create a .sops.yaml in your repo root
cat > .sops.yaml << 'EOF'
creation_rules:
  - path_regex: .production.
    kms: "arn:aws:kms:us-east-1:123456789012:key/prod-key-id"
  - path_regex: .staging.
    kms: "arn:aws:kms:us-east-1:123456789012:key/staging-key-id"
EOF

# Create and encrypt a secrets file
sops secrets/production/api.yaml

# secrets/production/api.yaml (after encryption)
apiVersion: v1
kind: Secret
metadata:
  name: api-secrets
type: Opaque
stringData:
  DB_PASSWORD: ENC[AES256_GCM,data:s3cur3...,type:str]
  API_KEY: ENC[AES256_GCM,data:sk-prod...,type:str]
sops:
  kms:
    - arn: arn:aws:kms:us-east-1:123456789012:key/prod-key-id
      created_at: "2026-06-17T10:00:00Z"
      enc: AQICAHh...
  version: 3.7.3

Decrypting in CI/CD

# In your deployment pipeline
sops -d secrets/production/api.yaml | kubectl apply -f -

SOPS With Flux (GitOps)

# flux-system/kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: production-secrets
  namespace: flux-system
spec:
  interval: 10m
  path: ./secrets/production
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  decryption:
    provider: sops
    secretRef:
      name: sops-kms

Injecting Secrets Into Kubernetes Pods

External Secrets Operator (Works With Any Backend)

# SecretStore connects to your secrets backend
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa

---
# ExternalSecret syncs a specific secret
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-database
  namespace: production
spec:
  refreshInterval: 5m
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: api-database-secret
    creationPolicy: Owner
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: production/api/database
        property: password
    - secretKey: DB_USERNAME
      remoteRef:
        key: production/api/database
        property: username

Using the Secret in a Pod

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
spec:
  template:
    spec:
      containers:
        - name: api
          image: myregistry/api:v2.3.1
          envFrom:
            - secretRef:
                name: api-database-secret
          # OR mount as files
          volumeMounts:
            - name: secrets
              mountPath: /etc/secrets
              readOnly: true
      volumes:
        - name: secrets
          secret:
            secretName: api-database-secret

Secrets in CI/CD Pipelines

GitHub Actions

# .github/workflows/deploy.yml
jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write  # For OIDC
      contents: read
    steps:
      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-deploy
          aws-region: us-east-1

      # Secrets are never in the repo — fetched at runtime
      - name: Get secrets
        uses: aws-actions/aws-secretsmanager-get-secrets@v2
        with:
          secret-ids: |
            DB,production/api/database
          parse-json-secrets: true

      - name: Deploy
        run: |
          # DB_PASSWORD, DB_USERNAME are now in env
          echo "Deploying with secrets injected..."

Pre-Commit Hook to Prevent Leaks

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.0
    hooks:
      - id: gitleaks

  - repo: https://github.com/awslabs/git-secrets
    rev: master
    hooks:
      - id: git-secrets

# Install and configure
brew install gitleaks
gitleaks detect --source . --verbose

Comparison Matrix

Final Thought

The right tool depends on your environment. AWS-only shops: start with Secrets Manager and External Secrets Operator. Multi-cloud or need dynamic credentials: Vault is worth the operational overhead. GitOps with Flux/ArgoCD: SOPS gives you encrypted secrets in git without a separate infrastructure dependency.

Whatever you choose, the non-negotiable principle is: secrets never live in code, never in plaintext, and always with an audit trail. Set up pre-commit hooks today. You'll thank yourself the first time they catch an accidentally staged .env file.

---

Frequently Asked Questions

What is secrets management and why is it important?

Secrets management is the practice of securely storing, distributing, rotating, and auditing sensitive credentials like API keys, passwords, certificates, and tokens. It's important because hardcoded secrets in code are the leading cause of credential leaks. Proper secrets management prevents unauthorized access, enables automated rotation, and provides audit trails for compliance.

What is the difference between HashiCorp Vault and AWS Secrets Manager?

Vault is cloud-agnostic, supports dynamic secret generation, and offers advanced features like encryption as a service and SSH certificate authority. AWS Secrets Manager is simpler, fully managed, and integrates natively with AWS services (RDS automatic rotation, ECS/Lambda injection). Use Vault for multi-cloud or complex requirements; use AWS Secrets Manager for AWS-native workloads with simpler needs.

How do I rotate secrets without causing downtime?

Implement dual-credential support where your application accepts both old and new credentials simultaneously. Generate the new secret, deploy it alongside the old one, verify the new credential works, then revoke the old one. Use tools like Vault's dynamic secrets to generate short-lived credentials that don't need manual rotation at all.

How do I prevent secrets from being committed to Git?

Use pre-commit hooks with tools like gitleaks, detect-secrets, or trufflehog to scan for secrets before code is committed. Add .env files to .gitignore, use secret scanning features in GitHub/GitLab, and educate developers on using environment variables or secret managers instead of hardcoding. Scan Git history regularly since deleted secrets remain in commit history.

What is the best way to inject secrets into containers?

Mount secrets as files (not environment variables) since env vars appear in process listings and crash dumps. Use Kubernetes Secrets mounted as volumes, AWS Secrets Manager with sidecar injection, or Vault's agent injector. The application reads secrets from the mounted file path at startup. For Kubernetes, the External Secrets Operator syncs cloud secrets into Kubernetes Secret objects automatically.

---

Related Resources

• Base64 Encoder/Decoder — Base64 encoder for Kubernetes secrets
• JWT Token Decoder — JWT token decoder
• Production Reference Architectures — Zero-trust Kubernetes architecture
• Kubernetes ConfigMaps and Secrets — Managing secrets in Kubernetes
• AWS IAM Least Privilege Guide — Least privilege for secrets access