# 5 Production Outage Case Studies: Root Causes, Timelines & Lessons Learned
Every production outage teaches you something you couldn't have learned from a textbook. These are the stories that get told at 2 AM in war rooms, over coffee the next morning, and eventually in blameless post-mortems. They're the incidents that shaped how teams think about reliability, monitoring, and the fragility of distributed systems.
This post covers five real-world production incidents — composites drawn from actual outages across the industry. Each one follows the same structure: what happened, how we detected it, how we fixed it, and most importantly, how we made sure it never happened again.
If you're a junior engineer, consider this your field guide to the kinds of failures you'll inevitably face. If you're a senior engineer, you'll probably recognize a few of these patterns from your own battle scars.
---
Case Study 1: DNS TTL Cache Poisoning Causes Global Outage
Severity: SEV-1 (Complete service unavailability)
Duration: 4 hours 37 minutes
Affected Users: 100% of global traffic
Services Affected: All customer-facing applications
The Scenario
The platform engineering team was executing a planned CDN migration from Provider A to Provider B. The migration plan was straightforward: update DNS CNAME records to point from the old CDN to the new one, wait for propagation, verify traffic flow, and decommission the old CDN endpoints.
What nobody accounted for was that the old DNS records had been configured with a TTL of 86400 seconds — that's 24 hours. And several major ISP recursive resolvers were known to cache records beyond their stated TTL in violation of RFC standards.
Timeline
T+0 (14:00 UTC) — DNS records updated. CNAME changed from cdn-old.provider-a.net to cdn-new.provider-b.net. The team verified the change in their authoritative DNS provider's dashboard.
T+2h (16:00 UTC) — Old CDN endpoints decommissioned. The team observed that approximately 70% of traffic had shifted to the new CDN based on Provider B's analytics dashboard. They assumed the remaining 30% would catch up as caches expired.
T+3h (17:00 UTC) — Provider A completes decommissioning. The old CDN endpoints begin returning 503 errors instead of serving cached content.
T+3h 15m (17:15 UTC) — PagerDuty fires. Error rates spike from 0.1% to 34% globally. The on-call SRE is paged.
T+3h 30m (17:30 UTC) — Incident declared SEV-1. War room opened. Initial hypothesis: new CDN misconfiguration.
T+4h (18:00 UTC) — Root cause identified. dig commands from various global vantage points reveal that many resolvers are still returning the old CNAME. The old endpoints are gone, so those users hit a dead end.
T+4h 20m (18:20 UTC) — Emergency mitigation begins. The team contacts Provider A to restore the old endpoints as a pass-through proxy to the new CDN.
T+4h 37m (18:37 UTC) — Provider A restores endpoints. Traffic begins recovering. Full recovery confirmed within 15 minutes as the proxy warms up.
Root Cause Analysis
The root cause was a combination of three failures:
The migration plan had no validation gate between "DNS updated" and "old infra decommissioned." It treated DNS propagation as a deterministic event rather than the probabilistic, eventually-consistent process it actually is.
Impact
- 34% of global users experienced complete service unavailability for approximately 1 hour and 37 minutes.
- Estimated revenue loss: $2.3M
- Customer trust impact: 847 support tickets, 12 enterprise customer escalations
- SLA breach for 3 enterprise contracts
Prevention & Action Items
dnscheck, ThousandEyes, or Catchpoint can verify resolution from hundreds of vantage points worldwide.# Pre-migration: Lower TTL 48 hours before the change
# Check current TTL
dig +noall +answer example.com CNAME
# Monitor propagation from multiple vantage points
for resolver in 8.8.8.8 1.1.1.1 208.67.222.222 9.9.9.9; do
echo "=== $resolver ==="
dig @$resolver +short example.com CNAME
done
# Monitor traffic to old endpoints (don't decommission until this hits zero)
watch -n 30 'curl -s https://cdn-old.provider-a.net/health | jq .request_count'
---
Case Study 2: Database Connection Pool Exhaustion
Severity: SEV-1 (Complete application unavailability)
Duration: 2 hours 12 minutes
Affected Users: 100% of authenticated users
Services Affected: All services requiring database access
The Scenario
It was Black Friday. The e-commerce platform had been load-tested to handle 3x normal traffic. Marketing had launched a flash sale, and traffic hit 4.2x normal within the first 15 minutes. The application didn't crash from CPU or memory — it died because every single database connection was checked out and never returned.
The application used a connection pool configured with max_connections: 20 per instance, with 12 application instances running. That's 240 total connections against a PostgreSQL server configured with max_connections: 300. Under normal load, connections were checked out for 5-50ms and returned. Under the traffic spike, a handful of slow queries (a reporting query that leaked into the OLTP path) started taking 30+ seconds, holding connections hostage.
Symptoms
- Application health checks passing (they didn't check database connectivity)
- HTTP 500 errors spiking across all endpoints requiring auth
- Logs flooded with:
PG::ConnectionBad: could not obtain connection within 5.000 seconds - Database CPU and memory looked healthy (the DB wasn't overloaded — it was the pool that was saturated)
Investigation Steps
Step 1: On-call checked application dashboards. CPU, memory, and network all normal. Error rate at 89%.
Step 2: Checked database dashboards. PostgreSQL CPU at 40%, memory at 60%. Connections: 298/300. That's the smoking gun.
Step 3: Ran SELECT * FROM pg_stat_activity WHERE state != 'idle'; — found 47 connections executing the same reporting query with a LIKE '%keyword%' full table scan.
Step 4: Identified that a product recommendation service was calling an internal API that triggered a reporting query on every request instead of using cached results.
Resolution
Immediate: Killed the long-running reporting queries with SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE query LIKE '%reporting_query%' AND state = 'active';
Short-term: Blocked the recommendation service's access to the reporting endpoint via network policy. Restarted application instances to reset connection pools.
Long-term: Separated OLTP and OLAP workloads onto different database instances.
How Connection Pooling Works (And What Went Wrong)
A connection pool maintains a set of pre-established database connections that application threads can "borrow" and return. This avoids the overhead of establishing a new TCP connection and performing authentication for every query.
The failure mode here was connection starvation: slow queries held connections for 30+ seconds instead of the expected 5-50ms. With a pool size of 20 and queries taking 30 seconds, a single instance could only serve 0.67 queries per second instead of the expected 400+. Multiply by 12 instances, and the entire platform ground to a halt.
The critical insight: connection pool exhaustion often looks like a database problem but is actually an application-level resource management failure.
Prevention & Action Items
# pgbouncer.ini
[databases]
production = host=db-primary.internal port=5432 dbname=app
[pgbouncer]
pool_mode = transaction
max_client_conn = 1000
default_pool_size = 40
reserve_pool_size = 10
reserve_pool_timeout = 3
server_idle_timeout = 300
query_wait_timeout = 10
-- Per-role timeout for the application user
ALTER ROLE app_user SET statement_timeout = '5s';
-- Separate role for reporting with longer timeout
ALTER ROLE reporting_user SET statement_timeout = '60s';
---
Case Study 3: Cascading Failure from a Single Microservice
Severity: SEV-1 (Cascading multi-service outage)
Duration: 1 hour 48 minutes
Affected Users: 78% of all users
Services Affected: 14 of 23 microservices
The Scenario
The payments service deployed a new version that included an unintentional N+1 query bug. Under normal load, response times were acceptable — about 200ms. But during a traffic surge, the payments service P99 latency climbed to 12 seconds.
Here's where it gets ugly. The order service called the payments service synchronously with a 30-second timeout. When payments slowed down, order service threads piled up waiting for responses. The order service had 200 threads, and within minutes, all 200 were blocked waiting on payments.
Now the order service itself became unresponsive. The product catalog service called the order service to check inventory reservations. The notification service called the order service for order status. The API gateway's connection pool to the order service saturated. One slow service brought down 14 services through a chain of synchronous dependencies.
Timeline
T+0 (09:45 UTC) — Payments service deployment completes. All health checks pass.
T+22m (10:07 UTC) — Traffic increases during morning shopping peak. Payments service P99 climbs from 200ms to 4 seconds.
T+28m (10:13 UTC) — Order service error rate spikes to 23%. Thread pool exhaustion alerts fire.
T+31m (10:16 UTC) — API gateway starts returning 503 for all order-related endpoints.
T+35m (10:20 UTC) — Catalog service, notification service, and shipping service all report degradation. SEV-1 declared.
T+42m (10:27 UTC) — Incident commander identifies payments service as the origin point via distributed tracing (Jaeger).
T+55m (10:40 UTC) — Payments service rolled back to previous version.
T+1h 10m (10:55 UTC) — Payments service latency returns to normal, but downstream services still recovering due to backed-up request queues.
T+1h 48m (11:33 UTC) — All services report healthy. Queues drained. Full recovery confirmed.
How Circuit Breakers Should Have Prevented This
The architecture had circuit breakers configured — but with fatally permissive thresholds. The circuit breaker on the order→payments call was configured to trip after 50% error rate sustained for 60 seconds. Since payments wasn't returning errors (it was just slow), the circuit breaker never tripped. Timeouts aren't errors in the default configuration — they were classified as "slow successes."
A properly configured circuit breaker would have:
- Counted timeouts and slow responses as failures
- Tripped after 5 seconds of degraded performance, not 60
- Returned a fallback response (e.g., "payment processing queued") instead of blocking
Blast Radius
payments (origin) → order → catalog
→ notifications
→ shipping
→ recommendations
→ api-gateway → all frontend traffic
14 services impacted. 78% of user-facing functionality unavailable. The only services that survived were those with no direct or transitive dependency on the order service.
Prevention & Action Items
# Resilience4j circuit breaker configuration
resilience4j:
circuitbreaker:
instances:
paymentsService:
slidingWindowSize: 10
failureRateThreshold: 50
slowCallRateThreshold: 80
slowCallDurationThreshold: 2s
waitDurationInOpenState: 30s
permittedNumberOfCallsInHalfOpenState: 3
timelimiter:
instances:
paymentsService:
timeoutDuration: 3s
---
Case Study 4: TLS Certificate Expiry in Production
Severity: SEV-1 (All HTTPS endpoints unavailable)
Duration: 3 hours 14 minutes
Affected Users: 100% of all users
Services Affected: Every HTTPS endpoint (API, web app, admin panel, webhooks)
The Scenario
At 03:17 UTC on a Saturday morning, the wildcard TLS certificate for *.example.com expired. Every HTTPS connection was rejected by browsers and API clients. The platform was dead.
The team had cert-manager installed in their Kubernetes cluster. It had been working flawlessly for 18 months, automatically renewing Let's Encrypt certificates 30 days before expiry. So what went wrong?
Three months earlier, the team migrated their DNS from Route53 to Cloudflare. The cert-manager DNS-01 challenge solver was still configured to use Route53 credentials. When the renewal attempt fired 30 days before expiry, it failed silently. The cert-manager pod logged the error, but nobody was watching those logs. The certificate reached its expiry date without a valid renewal.
Why Automated Renewal Failed
The chain of failures:
Failed to create DNS01 challenge: route53: AccessDenied but there was no alert configured on certificate renewal failures.Emergency Response
T+0 (03:17 UTC) — Certificate expires. Monitoring detects 100% TLS handshake failures.
T+5m (03:22 UTC) — PagerDuty fires. On-call engineer wakes up and confirms all endpoints returning TLS errors.
T+25m (03:42 UTC) — On-call identifies expired certificate as the cause. Checks cert-manager logs, finds 30 days of renewal failure logs.
T+40m (03:57 UTC) — Attempts manual certificate renewal via cert-manager. Fails — same DNS solver issue.
T+55m (04:12 UTC) — Escalation. Senior engineer joins. Decision: manually obtain a certificate using certbot with HTTP-01 challenge (requires temporarily exposing port 80 without TLS).
T+1h 20m (04:37 UTC) — certbot certonly --standalone -d "*.example.com" -d "example.com" — fails. HTTP-01 doesn't support wildcard certificates.
T+1h 35m (04:52 UTC) — Team switches to DNS-01 with Cloudflare. Updates cert-manager ClusterIssuer with Cloudflare API credentials. Triggers manual renewal.
T+2h 10m (05:27 UTC) — New certificate issued by Let's Encrypt. Deployed to ingress controller.
T+2h 15m (05:32 UTC) — HTTPS endpoints recovering. Some edge caches still serving stale TLS errors.
T+3h 14m (06:31 UTC) — Full recovery confirmed across all regions.
Prevention & Action Items
# Prometheus alert rule for certificate expiry
groups:
- name: tls-certificates
rules:
- alert: CertificateExpiringIn30Days
expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < 2592000
for: 1h
labels:
severity: warning
annotations:
summary: "Certificate {{ $labels.name }} expires in less than 30 days"
- alert: CertificateExpiringIn7Days
expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < 604800
for: 1h
labels:
severity: critical
annotations:
summary: "Certificate {{ $labels.name }} expires in less than 7 days"
- alert: CertificateRenewalFailed
expr: certmanager_certificate_ready_status{condition="False"} == 1
for: 1h
labels:
severity: critical
annotations:
summary: "Certificate {{ $labels.name }} renewal has failed"
Certificate status conditions in your monitoring. A certificate in Ready=False state is a ticking time bomb.# Quick check: when do your certificates expire?
kubectl get certificates -A -o custom-columns=\
NAMESPACE:.metadata.namespace,\
NAME:.metadata.name,\
READY:.status.conditions[0].status,\
EXPIRY:.status.notAfter
# External check with openssl
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | \
openssl x509 -noout -dates
---
Case Study 5: Bad Deployment Causes Data Corruption
Severity: SEV-1 (Data integrity incident)
Duration: 5 hours 23 minutes (full remediation took 3 days)
Affected Users: 12% of users (those who interacted with the system during the incident window)
Services Affected: User profile service, billing service
The Scenario
The user profile team shipped a database schema migration that renamed a column: user_address became billing_address. The migration was part of a deployment that also updated the application code to use the new column name. Simple, right?
Here's what went wrong: the deployment used a rolling update strategy. The migration ran first (as an init container), renaming the column immediately. Then new pods started rolling out. For approximately 4 minutes, old application pods (expecting user_address) coexisted with a database schema that only had billing_address.
During those 4 minutes, the old pods threw errors on every write to the address field. But worse — the ORM's error handling had a bug. Instead of failing the transaction, it silently wrote NULL to the address field. 2,847 users had their billing addresses set to NULL during the 4-minute window. The billing service later attempted to charge those users and failed, generating incorrect invoices and refund requests.
Detection via Monitoring Anomalies
T+0 (11:00 UTC) — Deployment begins. Migration runs. Column renamed.
T+4m (11:04 UTC) — Rolling update completes. All pods running new code. Errors stop.
T+2h (13:00 UTC) — Billing service reports anomalous failure rate for invoice generation. 3.2% of invoices failing with "missing billing address."
T+2h 15m (13:15 UTC) — On-call investigates. Queries reveal 2,847 users with NULL billing addresses that previously had values.
T+2h 30m (13:30 UTC) — Correlation with the 11:00 deployment identified. Git blame on the migration reveals the destructive rename.
T+2h 45m (13:45 UTC) — SEV-1 declared. Data integrity incident.
Rollback Complexity with Stateful Changes
This is where the team hit the wall. You can't just rollback a column rename. If you rename billing_address back to user_address, the new application code (which is already running and working correctly) will break.
The options were:
Option A: Rollback code AND database. Roll back the application to the old version AND rename the column back. But 2 hours of new data has been written to billing_address. Rolling back the column means losing that data.
Option B: Forward-fix the data. Keep the new schema, restore the NULL'd addresses from backup, and fix the ORM bug. This is what the team chose.
Resolution steps:
Total remediation time: 5 hours 23 minutes for the immediate fix. Full cleanup (customer communications, refund processing, audit) took 3 additional days.
Prevention & Action Items
-- Phase 1: EXPAND (deploy first, no code changes yet)
ALTER TABLE users ADD COLUMN billing_address TEXT;
UPDATE users SET billing_address = user_address;
-- Phase 2: MIGRATE CODE (deploy application reading/writing both columns)
-- Application writes to BOTH columns during transition
-- Phase 3: CONTRACT (only after all code is using new column)
-- Wait at least one full deployment cycle
ALTER TABLE users DROP COLUMN user_address;
# Example CI check: reject destructive migrations
# .github/workflows/migration-check.yml
- name: Check for destructive migrations
run: |
if grep -iE "(DROP COLUMN|RENAME COLUMN|DROP TABLE)" db/migrations/*.sql; then
echo "::error::Destructive migration detected. Use expand-and-contract pattern."
exit 1
fi
-- Data integrity monitoring query (run every 5 minutes)
SELECT
'billing_address' as field,
COUNT(*) FILTER (WHERE billing_address IS NULL) as null_count,
COUNT(*) as total_count,
ROUND(100.0 COUNT() FILTER (WHERE billing_address IS NULL) / COUNT(*), 2) as null_percentage
FROM users
WHERE updated_at > NOW() - INTERVAL '10 minutes';
-- Alert if null_percentage spikes above baseline
---
Common Patterns Across All Five Incidents
Looking across these five case studies, several themes emerge:
1. Silent Failures Are the Deadliest
In every case, the system failed silently before it failed loudly. DNS cached stale records silently. Connection pools saturated without alerting. Circuit breakers didn't trip. Certificate renewal failed 30 days before the impact. The ORM wrote NULLs silently. The time between silent failure and visible impact is your detection gap — and it's where damage accumulates.
2. Monitoring the Happy Path Isn't Monitoring
Health checks that don't test database connectivity. Alerts that only fire on HTTP 500s (not timeouts). Certificate monitoring that only checks "is it currently valid?" instead of "will it still be valid next week?" Your monitoring is only as good as the failure modes it covers.
3. Rollback Must Be a First-Class Capability
Every deployment plan needs a rollback plan. For stateless changes, that's straightforward. For stateful changes (schema migrations, data transformations), rollback is architecturally complex and must be designed explicitly. If you can't articulate how to undo a change, you're not ready to deploy it.
4. Dependencies Are Liabilities
A service calling another service synchronously inherits that service's failure modes. A deployment that depends on DNS propagation inherits DNS's eventual consistency model. A certificate renewal that depends on a specific DNS provider inherits that provider's API availability. Every dependency is a failure mode you're importing.
5. Blameless Post-Mortems Drive Improvement
None of these incidents were caused by a single person making a mistake. They were all systemic failures — gaps in process, monitoring, architecture, or testing. Blame prevents learning. Root cause analysis enables prevention.
---
Building Your Incident Response Muscle
The best time to prepare for a production outage is before it happens. Here's a starter checklist:
- [ ] Run regular game day exercises simulating these failure modes
- [ ] Maintain runbooks for common failure scenarios (certificate renewal, database failover, rollback procedures)
- [ ] Implement the alerting rules from this post before you need them
- [ ] Practice your incident response communication (status pages, customer communication)
- [ ] Review your architecture for synchronous dependency chains
- [ ] Audit your schema migration strategy for expand-and-contract compliance
- [ ] Verify your monitoring covers silent failure modes, not just loud ones
Every outage is expensive. But an outage you learn from is an investment. The teams that thrive in production aren't the ones that never fail — they're the ones that fail less often, detect faster, recover quicker, and never make the same mistake twice.
---
Have your own war stories? The DevOpsKit community shares incident learnings regularly. Every post-mortem shared is an outage prevented for someone else.
---
Frequently Asked Questions
What are the most common causes of production outages?
The top causes are: deployment failures (bad configs pushed to prod), dependency failures (database, third-party API outages), resource exhaustion (memory leaks, disk full, connection pool exhaustion), network issues (DNS failures, certificate expiration), and cascading failures (retry storms, missing circuit breakers). Most outages stem from changes deployed without adequate testing.
How should I structure a post-incident review?
Include: timeline of events (when detected, escalated, mitigated, resolved), root cause analysis (5 whys technique), contributing factors, impact assessment (duration, users affected, revenue lost), what went well, what went poorly, and action items with owners and deadlines. Focus on systemic improvements, not blame. Share findings widely to prevent similar incidents.
What is MTTR and how do I reduce it?
MTTR (Mean Time To Recovery) is the average time from incident detection to resolution. Reduce it by: improving monitoring to detect issues faster, creating runbooks for common scenarios, implementing automated rollback capabilities, practicing incident response through game days, and reducing deployment size so changes are easier to reason about and revert.
How do I prevent cascading failures in distributed systems?
Implement circuit breakers to stop calling failing services, use bulkheads to isolate failures, set aggressive timeouts with exponential backoff on retries, and design graceful degradation paths. Add rate limiting to protect services from surge traffic during partial failures. Load test failure scenarios regularly to verify your resilience patterns work.