Eliminating Alert Fatigue — From 100+ Daily Alerts to SLO-Based Alerting

Q: TL;DR — Quick Fix

Stop alerting on causes (CPU > 80%, memory > 70%). Start alerting on symptoms (error rate > 1%, latency p99 > 500ms). - alert: HighCPU expr: node_cpu_usage > 0.8

Q: Why Alert Fatigue Kills On-Call Teams

ALERT FATIGUE CYCLE — WHY ON-CALL

Q: Restructuring Alert Severity

| Severity | Response | SLA | Route | Example | |----------|----------|-----|-------|---------| | Critical | Wake someone up | 5 min | PagerDuty | Error budget burning at 14x rate |

Q: Converting Existing Alerts to SLO-Based

For every existing alert, ask: 1. Does this directly indicate user impact? Keep and refine 2. Does this predict future user impact? Convert to ticket-level

Q: What's happening

Payment API error rate is burning error budget at 14.4x the sustainable rate.

Q: Impact

Users are experiencing payment failures. Revenue is directly affected.

Q: Immediate Actions

1. Check deployment history: kubectl rollout history deployment/payment-api 2. If recent deploy, rollback: kubectl rollout undo deployment/payment-api 3. Check upstream dependencies: curl -s http://payment-api:8080/health

TL;DR — Quick Fix

Stop alerting on causes (CPU > 80%, memory > 70%). Start alerting on symptoms (error rate > 1%, latency p99 > 500ms).

# Before: Cause-based (noisy, non-actionable)
alert: HighCPU
  expr: node_cpu_usage > 0.8
  # Fires 50 times/day. Is anything actually broken? Who knows.

# After: Symptom-based (actionable, user-impacting)
alert: HighErrorRate
  expr: |
    sum(rate(http_requests_total{status=~"5.."}[5m]))
    / sum(rate(http_requests_total[5m])) > 0.01
  for: 5m
  # Fires when users are actually experiencing errors.

---

Why Alert Fatigue Kills On-Call Teams

The core problem: Traditional threshold-based alerting answers "is a metric above X?" instead of "are users experiencing problems?"

The SLO Framework: Alert on What Matters

Define Your SLIs (Service Level Indicators)

SLIs are the metrics that directly measure user experience:

SLI Type	What it measures	Example
Availability	Success rate of requests	99.9% of requests return non-5xx
Latency	Request speed	99% of requests complete in < 300ms
Throughput	System capacity	Process > 1000 requests/sec
Correctness	Data accuracy	99.99% of writes confirmed within 1s

Set Your SLOs (Service Level Objectives)

service: payment-api
slos:
  - name: availability
    target: 99.9%      # 43.2 min downtime allowed per month
    window: 30d
    sli: |
      sum(rate(http_requests_total{service="payment-api",status!~"5.."}[5m]))
      / sum(rate(http_requests_total{service="payment-api"}[5m]))

  - name: latency
    target: 99%        # 1% of requests can be slow
    window: 30d
    sli: |
      histogram_quantile(0.99,
        sum(rate(http_request_duration_seconds_bucket{service="payment-api"}[5m])) by (le)
      ) < 0.3

Calculate Error Budgets

Error Budget = 1 - SLO Target

For 99.9% availability over 30 days:
  Error Budget = 0.1% = 43.2 minutes of downtime allowed
  
For 99% latency SLO:
  Error Budget = 1% = 1% of requests can be slow

Multi-Window Burn Rate Alerting

Instead of simple threshold alerts, use burn rate — how fast you're consuming your error budget:

groups:
  - name: slo-burn-rate
    rules:
      # Critical: burning 14.4x budget in last hour
      # Will exhaust 30-day budget in 2 days at this rate
      - alert: PaymentAPIHighBurnRate_Critical
        expr: |
          (
            sum(rate(http_requests_total{service="payment-api",status=~"5.."}[1h]))
            / sum(rate(http_requests_total{service="payment-api"}[1h]))
          ) > (14.4 * 0.001)
          and
          (
            sum(rate(http_requests_total{service="payment-api",status=~"5.."}[5m]))
            / sum(rate(http_requests_total{service="payment-api"}[5m]))
          ) > (14.4 * 0.001)
        for: 2m
        labels:
          severity: critical
          team: payments
        annotations:
          summary: "Payment API burning error budget at 14.4x rate"
          description: "At this rate, 30-day error budget exhausts in 2 days"
          runbook: "https://runbooks.internal/payment-api/high-error-rate"

      # Warning: burning 6x budget over 6 hours
      - alert: PaymentAPIHighBurnRate_Warning
        expr: |
          (
            sum(rate(http_requests_total{service="payment-api",status=~"5.."}[6h]))
            / sum(rate(http_requests_total{service="payment-api"}[6h]))
          ) > (6 * 0.001)
          and
          (
            sum(rate(http_requests_total{service="payment-api",status=~"5.."}[30m]))
            / sum(rate(http_requests_total{service="payment-api"}[30m]))
          ) > (6 * 0.001)
        for: 5m
        labels:
          severity: warning
          team: payments
        annotations:
          summary: "Payment API burning error budget at 6x rate"
          description: "At this rate, 30-day error budget exhausts in 5 days"

      # Slow burn: ticket-worthy, not page-worthy
      - alert: PaymentAPISlowBurn
        expr: |
          (
            sum(rate(http_requests_total{service="payment-api",status=~"5.."}[3d]))
            / sum(rate(http_requests_total{service="payment-api"}[3d]))
          ) > (3 * 0.001)
        for: 1h
        labels:
          severity: ticket
          team: payments
        annotations:
          summary: "Payment API slow budget burn — create ticket for investigation"

Why Multi-Window?

Using two time windows (long + short) prevents:

Long window only: Slow to fire during sudden spikes
Short window only: Fires on brief transient blips

Both windows must be true for the alert to fire.

Restructuring Alert Severity

Severity	Response	SLA	Route	Example
Critical	Wake someone up	5 min	PagerDuty	Error budget burning at 14x rate
Warning	Look at it during work hours	4 hours	Slack channel	Error budget burning at 6x rate
Ticket	Fix this sprint	1 week	Jira/Linear	Slow error budget drain
Info	Dashboard only	None	Grafana	CPU elevated but no user impact

Alert Routing Configuration

# Alertmanager config route: receiver: 'slack-info' group_by: ['alertname', 'service'] group_wait: 30s group_interval: 5m repeat_interval: 4h routes: - match: severity: critical receiver: 'pagerduty-critical' repeat_interval: 5m continue: true - match: severity: warning receiver: 'slack-warnings' repeat_interval: 1h - match: severity: ticket receiver: 'jira-tickets' repeat_interval: 24h receivers: - name: 'pagerduty-critical' pagerduty_configs: - service_key: '<key>' description: '{{ .CommonAnnotations.summary }}' details: runbook: '{{ .CommonAnnotations.runbook }}' - name: 'slack-warnings' slack_configs: - api_url: 'https://hooks.slack.com/xxx' channel: '#alerts-warnings' title: '{{ .CommonAnnotations.summary }}'

text: '{{ .CommonAnnotations.description }}'

Converting Existing Alerts to SLO-Based

Classify Each Alert

For every existing alert, ask:

Does this directly indicate user impact? Keep and refine

Does this predict future user impact? Convert to ticket-level

Is this just a system metric with no clear user impact? Delete or move to dashboard

Map Causes to Symptoms

Cause Alert (Delete)	Symptom Alert (Keep)
CPU > 80%	Request latency p99 > SLO threshold
Memory > 70%	OOMKilled rate increasing
Pod restarts > 3	Availability SLI below target
Disk > 85%	Write failures occurring
Goroutines > 10000	Request throughput dropping

Runbook Template for Every Alert

Every paging alert must have a linked runbook:

# Alert: PaymentAPIHighBurnRate_Critical

What&#39;s happening
Payment API error rate is burning error budget at 14.4x the sustainable rate.

Impact
Users are experiencing payment failures. Revenue is directly affected.

Immediate Actions
Check deployment history: kubectl rollout history deployment/payment-api
If recent deploy, rollback: kubectl rollout undo deployment/payment-api
Check upstream dependencies: curl -s http://payment-api:8080/health
Check database connectivity: kubectl exec -it payment-api-xxx -- pg_isready

Escalation
After 15 min without resolution: page backend-lead
After 30 min: page engineering-manager

Past Incidents
2026-07-15: Caused by expired DB connection pool (JIRA-1234)
2026-06-22: Caused by upstream Stripe rate limiting (JIRA-1189)

---

Frequently Asked Questions

What is an error budget?

An error budget is the maximum amount of unreliability your SLO allows. For a 99.9% availability SLO over 30 days, your error budget is 0.1% (about 43 minutes of downtime). When you consume your error budget, it signals that reliability needs investment before shipping new features.

How many alerts should a team have?

A healthy on-call rotation should have fewer than 2 pages per shift that require human intervention. If you're getting more than 5 actionable pages per week per service, your alerts are either too sensitive or your system has real reliability issues that need engineering investment.

What's the difference between threshold alerts and burn rate alerts?

Threshold alerts fire when a metric crosses a fixed value (CPU > 80%). Burn rate alerts fire when you're consuming your error budget faster than sustainable. Burn rate is proportional to both the severity and duration of the issue, which means brief spikes don't page but sustained degradation does.

Should I delete all my infrastructure alerts?

No, but restructure them. Infrastructure metrics (CPU, memory, disk) should feed into dashboards and ticket-level alerts, not pages. Only alert on infrastructure when it directly causes user-facing symptoms.

How do I get buy-in for reducing alerts?

Track these metrics for 2 weeks: alert volume per day, percentage of alerts that led to action, time spent investigating non-actionable alerts, and alert-to-incident ratio. Present the data showing that 90%+ of alerts are noise and propose a 2-week trial of SLO-based alerting for one service.

---

Monitoring & Alerting for Production Systems — Full observability stack setup
Datadog vs. New Relic vs. Prometheus — Choosing monitoring tools
Production Outage Case Studies — Real incident post-mortems
Troubleshooting Playbooks for Production — Structured incident response