Production Monitoring — Stop Getting Paged at 3 AM for Stuff That Doesn't Matter

You Can't Fix What You Can't See

I've been on-call for systems processing millions of requests per day. The difference between a 2-minute incident and a 2-hour incident almost always comes down to monitoring quality. Bad monitoring means you're grepping logs at 3am trying to figure out what broke. Good monitoring means your pager fires with context: what failed, since when, and what's impacted.

This guide covers what actually matters: the right metrics, the right alerts, and how to avoid the alert fatigue that makes teams ignore their pagers.

The 4 Golden Signals

Google's SRE book nailed it. If you monitor nothing else, monitor these:

1. Latency

The time it takes to serve a request. Track both successful and failed requests separately — a fast 500 error skews your p50 if lumped together.

# p99 latency for successful requests over 5 minutes
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket{status!~"5.."}[5m])) by (le, service)
)

# p50 latency by endpoint
histogram_quantile(0.50,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, handler)
)

2. Traffic

Request volume. How much demand is on your system right now.

# Requests per second by service
sum(rate(http_requests_total[5m])) by (service)

# Compare current traffic to same time last week
sum(rate(http_requests_total[5m])) /
sum(rate(http_requests_total[5m] offset 7d))

3. Errors

The rate of failed requests. Both explicit (5xx) and implicit (200 with wrong content, timeouts treated as success).

# Error rate as percentage
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)
* 100

# Errors by type
sum(rate(http_requests_total{status=~"5.."}[5m])) by (status, handler)

4. Saturation

How "full" your service is. CPU, memory, disk, queue depth, connection pools.

# CPU saturation (throttling)
sum(rate(container_cpu_cfs_throttled_seconds_total[5m])) by (pod)
/
sum(rate(container_cpu_usage_seconds_total[5m])) by (pod)

# Memory saturation
container_memory_working_set_bytes / container_spec_memory_limit_bytes

# Disk I/O saturation
rate(node_disk_io_time_weighted_seconds_total[5m])

Setting Up Prometheus + Grafana on Kubernetes

Prometheus Deployment

Use the kube-prometheus-stack Helm chart. It bundles Prometheus, Grafana, Alertmanager, and pre-built dashboards:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.retention=30d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=100Gi \
  --set grafana.adminPassword=changeme \
  --values custom-values.yaml

Custom Values for Production

# custom-values.yaml
prometheus:
  prometheusSpec:
    retention: 30d
    retentionSize: "90GB"
    resources:
      requests:
        memory: 4Gi
        cpu: "2"
      limits:
        memory: 8Gi
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: gp3
          resources:
            requests:
              storage: 100Gi
    # Scrape every 15s for production
    scrapeInterval: 15s
    evaluationInterval: 15s

alertmanager:
  config:
    route:
      receiver: 'slack-critical'
      group_by: ['alertname', 'namespace', 'service']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      routes:
        - match:
            severity: critical
          receiver: 'pagerduty-critical'
          repeat_interval: 5m
        - match:
            severity: warning
          receiver: 'slack-warnings'
          repeat_interval: 1h
    receivers:
      - name: 'pagerduty-critical'
        pagerduty_configs:
          - service_key_file: /etc/alertmanager/secrets/pagerduty-key
      - name: 'slack-critical'
        slack_configs:
          - api_url_file: /etc/alertmanager/secrets/slack-webhook
            channel: '#incidents'
            title: '{{ .GroupLabels.alertname }}'
            text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'
      - name: 'slack-warnings'
        slack_configs:
          - api_url_file: /etc/alertmanager/secrets/slack-webhook
            channel: '#monitoring'

grafana:
  resources:
    requests:
      memory: 512Mi
      cpu: 250m
  persistence:
    enabled: true
    size: 10Gi

ServiceMonitor for Your Application

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: api-service
  namespace: monitoring
  labels:
    release: monitoring
spec:
  namespaceSelector:
    matchNames:
      - production
  selector:
    matchLabels:
      app: api-service
  endpoints:
    - port: metrics
      interval: 15s
      path: /metrics

SLOs vs SLAs: Know the Difference

• SLI (Service Level Indicator): The metric itself. "99.2% of requests returned 2xx in the last 30 days."
• SLO (Service Level Objective): Your internal target. "We aim for 99.9% availability."
• SLA (Service Level Agreement): The contractual obligation. "We guarantee 99.5% uptime or we pay credits."

Your SLO should always be stricter than your SLA. If your SLA is 99.5%, target 99.9% internally. The gap is your error budget — room to deploy, experiment, and recover without breaching contracts.

Defining an Error Budget

# Calculate remaining error budget for the month
# SLO: 99.9% availability = 0.1% error budget = ~43 minutes/month

# Total requests this month
sum(increase(http_requests_total[30d]))

# Failed requests this month
sum(increase(http_requests_total{status=~"5.."}[30d]))

# Error budget remaining (as percentage of budget consumed)
(
  sum(increase(http_requests_total{status=~"5.."}[30d]))
  /
  (sum(increase(http_requests_total[30d])) * 0.001)
) * 100

SLO-Based Alert (Burn Rate)

Alert when you're burning through your error budget too fast:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: slo-alerts
  namespace: monitoring
spec:
  groups:
    - name: slo.rules
      rules:
        # Fast burn: 14.4x burn rate over 1 hour (pages immediately)
        - alert: HighErrorBurnRate
          expr: |
            (
              sum(rate(http_requests_total{status=~"5.."}[1h]))
              /
              sum(rate(http_requests_total[1h]))
            ) > (14.4 * 0.001)
          for: 2m
          labels:
            severity: critical
          annotations:
            summary: "Error budget burning 14.4x faster than sustainable"
            description: "At this rate, the entire monthly error budget will be consumed in 2 days."

        # Slow burn: 3x burn rate over 6 hours (tickets, not pages)
        - alert: ElevatedErrorBurnRate
          expr: |
            (
              sum(rate(http_requests_total{status=~"5.."}[6h]))
              /
              sum(rate(http_requests_total[6h]))
            ) > (3 * 0.001)
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "Error budget burning 3x faster than sustainable"
            description: "Investigate during business hours. Budget will be exhausted in 10 days at this rate."

Preventing Alert Fatigue

Alert fatigue kills incident response. When everything pages, nothing pages. Here's how to keep your alerts meaningful:

Rules for Good Alerts

Alert on symptoms, not causes. Alert on "users are seeing errors," not "CPU is at 80%." High CPU with happy users isn't an incident.

Every alert must be actionable. If the on-call engineer can't do anything about it, it's not an alert — it's a dashboard metric.

Use multiple severity levels. Critical = pages. Warning = Slack. Info = dashboard only.

Set proper for durations. A 5-second CPU spike isn't an alert. Sustained for 5+ minutes is.

Group related alerts. Don't fire 50 alerts for one incident. Alertmanager's group_by is your friend.

Bad Alert vs Good Alert

# BAD: Fires constantly, not actionable
alert: HighCPU
  expr: node_cpu_seconds_total > 0.8

  for: 1m
  labels:
    severity: critical

# GOOD: User-facing impact, with context
alert: APIHighLatency
  expr: |

    histogram_quantile(0.99,
      sum(rate(http_request_duration_seconds_bucket{service="api"}[5m])) by (le)
    ) > 2.0
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "API p99 latency exceeds 2s for 5+ minutes"
    runbook: "https://wiki.internal/runbooks/api-high-latency"
    dashboard: "https://grafana.internal/d/api-overview"

Essential PromQL Queries for Your Dashboard

# Request rate with comparison to yesterday
sum(rate(http_requests_total[5m]))
sum(rate(http_requests_total[5m] offset 1d))

# Top 5 slowest endpoints
topk(5,
  histogram_quantile(0.95,
    sum(rate(http_request_duration_seconds_bucket[5m])) by (le, handler)
  )
)

# Pod restart rate (indicator of crashes)
sum(increase(kube_pod_container_status_restarts_total[1h])) by (namespace, pod) > 0

# Disk will be full in 4 hours (predictive)
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4*3600) < 0

# Connection pool saturation
sum(hikaricp_connections_active) by (pool)
/
sum(hikaricp_connections_max) by (pool)

What to Put on Your Grafana Dashboard

Structure your dashboards in layers:

Overview dashboard: All services at a glance. RED metrics (Rate, Errors, Duration) per service. This is what you look at first during an incident.

Service dashboard: Deep dive into one service. All 4 golden signals, pod-level metrics, dependency health.

Infrastructure dashboard: Node CPU/memory/disk/network. Useful for capacity planning, not incident response.

The overview dashboard should answer one question in under 10 seconds: "Is anything broken right now, and where?"

Final Thought

Good monitoring is an investment that pays off at 3am. Spend the time upfront to instrument properly, define meaningful SLOs, and keep your alert count low. The goal isn't more data — it's faster understanding. When your pager fires, you should know within 60 seconds what's broken, how bad it is, and where to start looking.

---

Frequently Asked Questions

What metrics should I monitor for production systems?

Focus on the four golden signals: latency (response time), traffic (requests per second), errors (error rate/percentage), and saturation (resource utilization). Also monitor infrastructure metrics (CPU, memory, disk I/O, network) and business metrics (orders/minute, signups). Alert on symptoms (high error rate) rather than causes (high CPU) when possible.

How do I reduce alert fatigue in my team?

Implement alert levels: pages for customer-facing impact requiring immediate action, tickets for issues that need attention within hours, and logs for informational items. Remove alerts that are never actionable, set appropriate thresholds above normal variation, and use alert grouping/deduplication. Every alert should have a clear runbook documenting what to do.

What is the difference between monitoring, observability, and alerting?

Monitoring tracks predefined metrics to detect known failure modes. Observability is the ability to understand system state from external outputs (metrics, logs, traces) even for unknown failures. Alerting is the notification mechanism triggered when metrics cross thresholds. You need all three: observability to debug, monitoring for baselines, and alerting for response.

How do I set up effective alerting thresholds?

Base thresholds on historical data — set them at 2-3 standard deviations above the normal baseline, or use percentile-based alerts (P99 latency > 500ms). Implement multi-window alerting (5-minute and 1-hour windows) to catch both sudden spikes and gradual degradation. Start loose and tighten over time rather than alerting on everything from day one.

---

Related Resources

• SLA Calculator — SLA and error budget calculator
• DORA Metrics Calculator — DORA metrics calculator
• Production Troubleshooting Scenarios — Production troubleshooting scenarios
• Zero Downtime Deployment Strategies — Monitoring during deployments
• Linux Performance Troubleshooting — Diagnosing performance issues that trigger alerts