TL;DR Quick Fix
Stop spending weeks gathering compliance evidence manually. Automate it with Steampipe:
# Install Steampipe with AWS plugin
brew install turbot/tap/steampipe
steampipe plugin install aws
# Run a compliance check immediately
steampipe check benchmark.cis_v150
# Export evidence as JSON for auditors
steampipe check benchmark.cis_v150 --export=evidence-$(date +%Y%m%d).json
For continuous enforcement, deploy CloudCustodian policies that auto-remediate non-compliant resources:
# Run a custodian policy to find unencrypted S3 buckets
custodian run -s output/ policies/s3-encryption.yml
---
Architecture Overview
---
The Pain of Manual Compliance
Traditional evidence collection involves:
- Screenshotting console settings manually
- Exporting CSV reports from each AWS account
- Asking team leads to confirm policies via email
- Spending 2-4 weeks preparing for each audit cycle
All of this is automatable. Here is how.
---
Steampipe for Cloud Resource Querying
SOC2 Access Control Evidence
-- steampipe query: List all IAM users with console access and MFA status
select
user_name,
password_enabled,
mfa_active,
password_last_used,
create_date
from
aws_iam_user
where
password_enabled = true
order by
mfa_active asc, password_last_used desc;
-- Find S3 buckets without encryption (SOC2 CC6.1)
select
name,
region,
server_side_encryption_configuration,
acl ->> 'Owner' as owner
from
aws_s3_bucket
where
server_side_encryption_configuration is null;
Running CIS Benchmarks
# Run full CIS AWS Foundations Benchmark v1.5.0
steampipe check benchmark.cis_v150 --export=cis-report.json --export=cis-report.html
# Run specific SOC2 controls
steampipe check benchmark.soc2_cc6 --export=soc2-cc6-evidence.json
# Run against multiple AWS accounts
steampipe check benchmark.cis_v150 \
--search-path-prefix=aws_prod,aws_staging,aws_dev
Custom Compliance Queries
-- ISO27001 A.12.4.1: Event logging evidence
-- Show CloudTrail status across all regions
select
name,
region,
is_multi_region_trail,
is_logging,
log_file_validation_enabled,
s3_bucket_name
from
aws_cloudtrail_trail
where
region = home_region;
-- SOC2 CC7.2: System monitoring
-- Verify GuardDuty is enabled in all regions
select
region,
status,
finding_publishing_frequency,
updated_at
from
aws_guardduty_detector;
---
CloudCustodian for Policy Enforcement
Policy Definitions
# policies/s3-encryption.yml
policies:
- name: s3-require-encryption
resource: s3
description: "SOC2 CC6.1 - All S3 buckets must have encryption"
filters:
- type: bucket-encryption
state: false
actions:
- type: set-bucket-encryption
crypto: AES256
- type: notify
template: default
to:
- security-team@company.com
transport:
type: sns
topic: arn:aws:sns:us-east-1:123456789012:compliance-alerts
- name: ec2-require-imdsv2
resource: ec2
description: "Require IMDSv2 on all EC2 instances"
filters:
- type: value
key: MetadataOptions.HttpTokens
value: optional
actions:
- type: modify-metadata-options
HttpTokens: required
- name: rds-require-encryption
resource: rds
description: "SOC2 CC6.1 - All RDS instances must be encrypted"
filters:
- StorageEncrypted: false
actions:
- type: tag
tags:
compliance-violation: "unencrypted-storage"
- type: notify
template: default
to:
- security-team@company.com
transport:
type: sns
topic: arn:aws:sns:us-east-1:123456789012:compliance-alerts
Running Custodian Policies
#!/bin/bash
# run-compliance-checks.sh
set -euo pipefail
POLICIES_DIR="./policies"
OUTPUT_DIR="./compliance-output/$(date +%Y%m%d)"
mkdir -p "$OUTPUT_DIR"
echo "Running CloudCustodian compliance checks..."
# Run all policies
for policy_file in "$POLICIES_DIR"/*.yml; do
echo "Processing: $policy_file"
custodian run -s "$OUTPUT_DIR" "$policy_file"
done
# Generate summary report
custodian report -s "$OUTPUT_DIR" --format=csv > "$OUTPUT_DIR/summary.csv"
# Upload evidence to S3
aws s3 sync "$OUTPUT_DIR" "s3://compliance-evidence/$(date +%Y%m%d)/"
echo "Compliance check complete. Results in: $OUTPUT_DIR"
---
Automated Evidence Collection Pipeline
# .github/workflows/compliance-evidence.yml
name: Compliance Evidence Collection
on:
schedule:
- cron: '0 6 *' # Daily at 6 AM UTC
workflow_dispatch:
permissions:
id-token: write
contents: read
jobs:
collect-evidence:
runs-on: ubuntu-latest
strategy:
matrix:
account: [production, staging, development]
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::${{ secrets[format('{0}_ACCOUNT_ID', matrix.account)] }}:role/compliance-reader
aws-region: us-east-1
- name: Install Steampipe
run: |
sudo /bin/sh -c "$(curl -fsSL https://steampipe.io/install/steampipe.sh)"
steampipe plugin install aws
- name: Run CIS Benchmark
run: |
steampipe check benchmark.cis_v150 \
--export=evidence/cis-${{ matrix.account }}.json \
--export=evidence/cis-${{ matrix.account }}.html
- name: Run SOC2 checks
run: |
steampipe check benchmark.soc2 \
--export=evidence/soc2-${{ matrix.account }}.json
- name: Upload evidence to S3
run: |
aws s3 sync evidence/ \
s3://compliance-evidence/$(date +%Y/%m/%d)/${{ matrix.account }}/
---
Mapping Controls to Infrastructure Checks
# compliance-mapping.yml
controls:
SOC2-CC6.1:
title: "Logical and Physical Access Controls"
checks:
- steampipe: "benchmark.cis_v150_1" # IAM
- custodian: "s3-require-encryption"
- custodian: "rds-require-encryption"
- aws_config: "encrypted-volumes"
SOC2-CC6.6:
title: "Security Boundaries"
checks:
- steampipe: "benchmark.cis_v150_4" # Networking
- custodian: "sg-restrict-ingress"
- aws_config: "vpc-flow-logs-enabled"
ISO27001-A.12.4.1:
title: "Event Logging"
checks:
- steampipe: "select * from aws_cloudtrail_trail where is_logging"
- custodian: "cloudtrail-enabled-all-regions"
- aws_config: "cloud-trail-enabled"
ISO27001-A.10.1.1:
title: "Cryptographic Controls"
checks:
- steampipe: "benchmark.cis_v150_2" # Storage
- custodian: "kms-key-rotation"
- aws_config: "cmk-backing-key-rotation-enabled"
---
Continuous Compliance Dashboard
# Export compliance metrics to Prometheus/Grafana
steampipe check benchmark.cis_v150 --export=json | python3 -c "
import json, sys
from prometheus_client import CollectorRegistry, Gauge, push_to_gateway
data = json.load(sys.stdin)
registry = CollectorRegistry()
compliance_score = Gauge('compliance_score_percent', 'Overall compliance score',
['framework', 'account'], registry=registry)
controls_passed = Gauge('compliance_controls_passed', 'Number of passed controls',
['framework', 'account'], registry=registry)
controls_failed = Gauge('compliance_controls_failed', 'Number of failed controls',
['framework', 'account'], registry=registry)
passed = data['summary']['passed']
failed = data['summary']['failed']
total = passed + failed
score = (passed / total * 100) if total > 0 else 0
compliance_score.labels('cis_v150', 'production').set(score)
controls_passed.labels('cis_v150', 'production').set(passed)
controls_failed.labels('cis_v150', 'production').set(failed)
push_to_gateway('pushgateway:9091', job='compliance', registry=registry)
"
---
FAQ
How often should I run compliance checks?
Daily for detection, real-time for enforcement. Run Steampipe benchmarks daily to generate evidence artifacts. Use CloudCustodian in real-time mode (via CloudTrail events) to catch and remediate violations as they happen. This gives auditors both point-in-time evidence and proof of continuous monitoring.
Can this replace a manual audit entirely?
No, but it dramatically reduces the manual effort (from weeks to hours). Auditors still need to verify that policies match the stated controls, interview staff about processes, and validate that automated checks are correctly implemented. Think of it as pre-packaging 80% of the evidence they need.
How do I handle multi-account AWS environments?
Use Steampipe aggregators to query across accounts, or run checks per account via CI matrix strategies (as shown above). CloudCustodian supports cross-account execution via STS assume-role. Centralize evidence in a dedicated compliance account S3 bucket.
What about compliance for Kubernetes workloads?
Use kube-bench for CIS Kubernetes benchmarks, and OPA/Gatekeeper for policy enforcement. Steampipe has a Kubernetes plugin that can query cluster resources with SQL. Map these checks to the same control framework alongside your cloud infrastructure checks.
How do I prove continuous compliance between audits?
Store daily evidence snapshots in S3 with lifecycle rules (keep for 3 years for SOC2). Your compliance dashboard should show a timeline of scores. When auditors ask about any date, you can point to the exact evidence collected that day.
---