TL;DR — Quick Fix
Use this post-mortem template immediately after any SEV1/SEV2 incident:
# Post-Mortem: [Incident Title]
Date: YYYY-MM-DD | Duration: X hours Y minutes
Severity: SEV1/SEV2 | Lead: [Name]
Status: Draft > Review > Final
Summary (2-3 sentences max)
What happened, how long it lasted, and what was impacted.
Impact
- Users affected: X
- Revenue impact: $Y
- SLA breach: Yes/No (remaining error budget: Z%)
Timeline (UTC)
Time Event 14:00 First alert fires 14:05 On-call acknowledges 14:15 Root cause identified 14:30 Fix deployed 14:45 Recovery confirmed
Root Cause
[One paragraph explaining the technical root cause]
5 Whys
Why did the outage occur? -> [Answer]
Why did [Answer 1] happen? -> [Answer]
Why did [Answer 2] happen? -> [Answer]
Why did [Answer 3] happen? -> [Answer]
Why did [Answer 4] happen? -> [Systemic root cause]
Action Items
Priority Action Owner Due Date Status P0 Fix immediate cause @name +1 day Done P1 Add missing alert @name +1 week In Progress
| P2 | Improve runbook | @name | +2 weeks | Todo |---
Architecture — Incident Lifecycle and Post-Mortem Flow
---
Step 1 — Severity Classification
| Severity | Criteria | Response Time | Post-Mortem Required |
|---|---|---|---|
| SEV1 | Complete outage, data loss risk | Immediate (< 5 min) | Yes (within 48h) |
| SEV2 | Major feature degraded, > 10% users | < 15 min | Yes (within 72h) |
| SEV3 | Minor degradation, workaround exists | < 1 hour | Optional |
| SEV4 | Cosmetic issue, no user impact | Next business day | No |
---
Step 2 — Timeline Construction
# Gather timeline data from these sources
sources:
- alertmanager/pagerduty history
- deployment logs (ArgoCD, GitHub Actions)
- chat transcripts (Slack incident channel)
- CloudWatch/Datadog metrics
- git log (recent changes)
# Timeline format (always UTC)
timeline:
- time: "2026-08-01 14:00 UTC"
event: "Error rate alert fires (> 5% 5xx responses)"
source: "PagerDuty"
- time: "2026-08-01 14:03 UTC"
event: "On-call engineer acknowledges alert"
source: "PagerDuty"
- time: "2026-08-01 14:05 UTC"
event: "Investigation begins — database connection pool saturated"
source: "Slack #incident-20260801"
- time: "2026-08-01 14:12 UTC"
event: "Identified: deployment at 13:45 added N+1 query"
source: "git log + APM traces"
- time: "2026-08-01 14:15 UTC"
event: "Decision: rollback deployment"
source: "Slack #incident-20260801"
- time: "2026-08-01 14:18 UTC"
event: "Rollback initiated via ArgoCD"
source: "ArgoCD audit log"
- time: "2026-08-01 14:25 UTC"
event: "Rollback complete, error rate returning to baseline"
source: "Grafana dashboard"
- time: "2026-08-01 14:35 UTC"
event: "All clear — incident resolved (duration: 35 min)"
source: "PagerDuty resolved"
---
Step 3 — Root Cause Analysis (5 Whys)
## 5 Whys Analysis
Why did the outage occur?
Database connection pool was exhausted, causing 503 errors.
Why was the connection pool exhausted?
A new API endpoint executed 50 queries per request (N+1 problem).
Why did an N+1 query reach production?
No query count assertions in integration tests for this endpoint.
Why were there no query count tests?
Our testing guidelines do not include performance regression checks.
Why do our guidelines miss performance checks?
We optimized onboarding speed over production safety in test requirements.
Systemic Root Cause: Testing standards lack performance regression coverage.
Contributing Factors:
- No query-count linting in CI pipeline
- Database connection pool sized for normal load, no circuit breaker
- Deployment happened during peak traffic
---
Step 4 — Action Item Tracking
# action-items.yaml — tracked in project management tool
action_items:
- id: AI-001
priority: P0
action: "Roll forward with fix for N+1 query"
owner: "@backend-dev"
due: "2026-08-02"
status: "Done"
- id: AI-002
priority: P1
action: "Add query count assertions to integration tests"
owner: "@qa-lead"
due: "2026-08-08"
status: "In Progress"
- id: AI-003
priority: P1
action: "Add connection pool exhaustion alert"
owner: "@sre-team"
due: "2026-08-05"
status: "Done"
- id: AI-004
priority: P2
action: "Implement deployment windows during peak hours"
owner: "@platform-team"
due: "2026-08-15"
status: "Todo"
- id: AI-005
priority: P2
action: "Add database circuit breaker"
owner: "@backend-dev"
due: "2026-08-20"
status: "Todo"
---
Step 5 — Meeting Facilitation Guide
## Post-Mortem Meeting Agenda (45-60 minutes)
Ground Rules (read aloud at start)
This is a BLAMELESS review — we examine systems, not people
"Human error" is never a root cause — ask what made the error easy
Everyone involved is here to learn, not to defend
Action items need owners and deadlines before we leave
Agenda
Summary (facilitator, 2 min)
Timeline walkthrough (facilitator, 10 min)
Clarifying questions (all, 10 min)
Root cause discussion / 5 Whys (all, 15 min)
What went well (all, 5 min)
Action items brainstorm + assign (all, 10 min)
Wrap-up and next steps (facilitator, 3 min)
Anti-patterns to watch for:
- "Who did this?" -> "What allowed this to happen?"
- "They should have known" -> "How can we make this obvious?"
- "Just be more careful" -> "What guardrail prevents this?"
- Too many action items -> Limit to 3-5 high-impact items
---
Blameless Culture Principles
| Principle | What It Means | What It Does NOT Mean |
|---|---|---|
| Blameless | Examine systems and processes | No accountability |
| Learning-focused | Extract patterns to prevent recurrence | Ignore the incident |
| Transparent | Share broadly across organization | Name and shame |
| Action-oriented | Every post-mortem produces improvements | Endless meetings |
| Time-bounded | Complete within 72 hours | Rush without reflection |
---
Frequently Asked Questions
How do I get buy-in for blameless post-mortems from leadership?
Frame it as risk reduction: teams that fear blame hide information, leading to repeated incidents. Show data — companies with blameless cultures have faster recovery times and fewer repeat incidents. Start with one team as a pilot and share results.
What if the same person causes multiple incidents?
Blameless does not mean consequence-free. If a pattern emerges, it is a management conversation about training, support, or role fit — not a post-mortem finding. The post-mortem should focus on why the system allowed the same failure mode repeatedly.
How do we track action item completion?
Create tickets in your project management tool tagged with the incident ID. Review completion in weekly SRE syncs. Publish a monthly action item completion rate metric. Unresolved P0/P1 items older than 2 weeks should escalate automatically.
Should we post-mortem near-misses?
Yes. Near-misses are free learning without customer impact. They reveal the same systemic weaknesses that full outages do. Run lighter-weight incident reviews (20 minutes, written async) for near-misses and save full post-mortems for actual impact.
How detailed should the timeline be?
Include every event that helps explain what happened, with timestamps in UTC. Typical resolution: 2-5 minute intervals during active incident, with key decision points noted. Too much detail buries the signal; too little prevents learning.
---