How to Tame CloudWatch Log Ingestion Costs — Save 70% on Logging

TL;DR — Quick Fix

Find your most expensive log groups and apply immediate cost reduction:

# Find the top 10 most expensive log groups (by ingestion bytes)
aws logs describe-log-groups \
  --query 'logGroups | sort_by(@, &storedBytes) | reverse(@) | [0:10].{Name:logGroupName, StoredGB: storedBytes}' \
  --output table

# Set retention policy (default is NEVER EXPIRE = infinite cost)
aws logs put-retention-policy \
  --log-group-name /aws/lambda/my-function \
  --retention-in-days 14

# Switch to Infrequent Access log class (50% cheaper ingestion)
aws logs create-log-group \
  --log-group-name /app/my-service \
  --log-group-class INFREQUENT_ACCESS

# Bulk-set retention on ALL log groups without a policy
for lg in $(aws logs describe-log-groups \
  --query 'logGroups[?!retentionInDays].logGroupName' --output text); do
  aws logs put-retention-policy --log-group-name "$lg" --retention-in-days 30
done

---

Architecture — Log Cost Optimization Pipeline

---

Step 1 — Identify Expensive Log Groups

#!/bin/bash
# find-expensive-logs.sh — Identify where your money goes

echo "=== Top Log Groups by Stored Bytes ==="
aws logs describe-log-groups \
  --query 'logGroups | sort_by(@, &storedBytes) | reverse(@) | [0:15].{Name:logGroupName, GB:storedBytes, Retention:retentionInDays}' \
  --output table

echo "=== Log Groups WITHOUT retention policy (infinite cost) ==="
aws logs describe-log-groups \
  --query 'logGroups[?!retentionInDays].{Name:logGroupName, StoredGB:storedBytes}' \
  --output table

---

Step 2 — Filter Logs at Source with FluentBit

# fluent-bit-values.yaml (Helm chart for K8s DaemonSet)
config:
  inputs: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            cri
        Mem_Buf_Limit     50MB
        Skip_Long_Lines   On
        Refresh_Interval  10

  filters: |
    [FILTER]
        Name    grep
        Match   kube.*
        Exclude log /healthz|/readyz|/livez|GET \/ HTTP/

    [FILTER]
        Name    grep
        Match   kube.*
        Exclude log ^{"level":"debug"

    [FILTER]
        Name          throttle
        Match         kube.chatty-service
        Rate          1000
        Window        5
        Interval      1s

  outputs: |
    [OUTPUT]
        Name              cloudwatch_logs
        Match             kube.*
        region            us-east-1
        log_group_name    /app/${TAG[4]}
        log_stream_prefix ${HOSTNAME}-
        auto_create_group On
        log_retention_days 14

    [OUTPUT]
        Name              s3
        Match             kube.*
        region            us-east-1
        bucket            logs-archive-bucket
        total_file_size   50M
        upload_timeout    60s
        s3_key_format     /logs/$TAG/%Y/%m/%d/%H/$UUID.gz
        compression       gzip

---

Step 3 — Log Class Selection

---

Step 4 — S3 Lifecycle Archiving

# Terraform — S3 bucket for log archival with lifecycle rules
resource "aws_s3_bucket" "log_archive" {
  bucket = "company-log-archive"
}

resource "aws_s3_bucket_lifecycle_configuration" "log_lifecycle" {
  bucket = aws_s3_bucket.log_archive.id

  rule {
    id     = "transition-to-glacier"
    status = "Enabled"

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }
    transition {
      days          = 90
      storage_class = "GLACIER"
    }
    transition {
      days          = 365
      storage_class = "DEEP_ARCHIVE"
    }
    expiration {
      days = 2555  # 7 years for compliance
    }
  }
}

# Subscription filter to stream from CloudWatch to S3 via Firehose
resource "aws_cloudwatch_log_subscription_filter" "to_s3" {
  name            = "all-logs-to-s3"
  log_group_name  = "/app/my-service"
  filter_pattern  = ""
  destination_arn = aws_kinesis_firehose_delivery_stream.logs.arn
  role_arn        = aws_iam_role.cw_to_firehose.arn
}

---

Step 5 — Retention Policy Automation

#!/bin/bash
# set-retention-all-groups.sh — Apply retention policies at scale

declare -A RETENTION_MAP
RETENTION_MAP["/aws/lambda/"]=14
RETENTION_MAP["/aws/ecs/"]=30
RETENTION_MAP["/app/"]=30
RETENTION_MAP["/aws/rds/"]=7
RETENTION_MAP["/aws/apigateway/"]=14

for lg in $(aws logs describe-log-groups \
  --query 'logGroups[?!retentionInDays].logGroupName' --output text); do
  retention=30  # default

  for prefix in "${!RETENTION_MAP[@]}"; do
    if [[ "$lg" == "$prefix"* ]]; then
      retention=${RETENTION_MAP[$prefix]}
      break
    fi
  done

  aws logs put-retention-policy \
    --log-group-name "$lg" \
    --retention-in-days $retention
  echo "Set ${retention}-day retention: $lg"
done

---

Cost Savings Summary

---

Frequently Asked Questions

What is the difference between Standard and Infrequent Access log class?

Standard class costs $0.50/GB ingestion and supports real-time log tailing, metric filters, and subscription filters. Infrequent Access costs $0.25/GB ingestion but does not support real-time tailing or metric filters. Use IA for logs you need to keep but rarely search.

Can I change the log class of an existing log group?

No, log class is immutable after creation. To switch, create a new log group with the desired class, update your logging configuration to point to it, and delete the old group once drained.

How do I query logs archived in S3?

Use Amazon Athena with a Glue crawler to query gzipped logs in S3. Create a table pointing to your S3 prefix, then run SQL queries. This costs approximately $5/TB scanned, which is far cheaper than CloudWatch Insights for large historical queries.

Is it safe to drop health check logs?

Yes. Health check logs (from ALB, Kubernetes probes, uptime monitors) typically account for 30-50% of log volume and provide zero debugging value. They only confirm the service was alive at that moment, which your monitoring system already tracks.

What about CloudWatch Logs Insights costs?

Insights charges $0.005 per GB scanned. If you scan 100GB of logs per query across a 30-day window, that is $0.50 per query. Reducing stored log volume through retention and filtering directly reduces Insights costs too.

---

Related Resources

• AWS NAT Gateway Cost Optimization
• Monitoring and Alerting for Production Systems
• Docker Container Logs Debugging
• Kubernetes Pod Troubleshooting Guide
• DevOps Infrastructure Sizing Guide