TL;DR — Quick Fix
Import this PromQL-based dashboard covering all 4 Golden Signals:
# Essential PromQL queries for the 4 signals:
# 1. Latency (p99)
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
# 2. Traffic (requests per second)
sum(rate(http_requests_total[5m])) by (handler)
# 3. Errors (error rate percentage)
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100
# 4. Saturation (memory utilization)
sum(container_memory_working_set_bytes{container!=""}) by (pod) /
sum(kube_pod_container_resource_limits{resource="memory"}) by (pod) * 100
---
The 4 Golden Signals
Google SRE's four golden signals tell you everything about service health.
---
PromQL Queries for Each Signal
Latency Panel
# Grafana panel — Latency percentiles
panels:
- title: "Request Latency (p50/p95/p99)"
type: timeseries
targets:
- expr: histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{job="$service"}[5m])) by (le))
legendFormat: "p50"
- expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="$service"}[5m])) by (le))
legendFormat: "p95"
- expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{job="$service"}[5m])) by (le))
legendFormat: "p99"
fieldConfig:
defaults:
unit: "s"
thresholds:
steps:
- { value: 0, color: green }
- { value: 0.5, color: yellow }
- { value: 1.0, color: red }
Traffic Panel
panels:
- title: "Request Rate by Status"
type: timeseries
targets:
- expr: sum(rate(http_requests_total{job="$service", status=~"2.."}[5m]))
legendFormat: "2xx Success"
- expr: sum(rate(http_requests_total{job="$service", status=~"4.."}[5m]))
legendFormat: "4xx Client"
- expr: sum(rate(http_requests_total{job="$service", status=~"5.."}[5m]))
legendFormat: "5xx Server"
fieldConfig:
defaults:
unit: "reqps"
Error Rate Panel
panels:
- title: "Error Rate % — SLO: 99.9%"
type: stat
targets:
- expr: |
(sum(rate(http_requests_total{job="$service", status=~"5.."}[5m]))
/ sum(rate(http_requests_total{job="$service"}[5m]))) * 100
fieldConfig:
defaults:
unit: "percent"
thresholds:
steps:
- { value: 0, color: green }
- { value: 0.1, color: yellow }
- { value: 1.0, color: red }
Saturation Panel
panels:
- title: "Pod Memory Saturation"
type: timeseries
targets:
- expr: |
sum(container_memory_working_set_bytes{namespace="$namespace", container!=""}) by (pod)
/ sum(kube_pod_container_resource_limits{namespace="$namespace", resource="memory"}) by (pod) * 100
legendFormat: "{{ pod }}"
fieldConfig:
defaults:
unit: "percent"
max: 100
---
USE Method for Infrastructure
The USE method (Utilization, Saturation, Errors) for infrastructure components:
# CPU Utilization per node
sum(rate(node_cpu_seconds_total{mode!="idle"}[5m])) by (instance) /
count(node_cpu_seconds_total{mode="idle"}) by (instance) * 100
# Disk I/O Saturation
rate(node_disk_io_time_weighted_seconds_total[5m])
# Network Errors
rate(node_network_receive_errs_total[5m]) + rate(node_network_transmit_errs_total[5m])
---
RED Method for Services
The RED method (Rate, Errors, Duration) for microservices:
# Rate — requests per second
sum(rate(http_requests_total{service="$service"}[5m]))
# Errors — failed requests per second
sum(rate(http_requests_total{service="$service", status=~"5.."}[5m]))
# Duration — request latency distribution
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service="$service"}[5m])) by (le))
---
Actionable vs Vanity Dashboards
# alerting-rules.yaml — Alert on Golden Signals
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: golden-signal-alerts
spec:
groups:
- name: golden-signals
rules:
- alert: HighLatencyP99
expr: |
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)) > 1.0
for: 5m
labels:
severity: warning
annotations:
summary: "{{ $labels.service }} p99 latency exceeds 1s"
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/ sum(rate(http_requests_total[5m])) by (service) > 0.01
for: 3m
labels:
severity: critical
annotations:
summary: "{{ $labels.service }} error rate above 1%"
- alert: TrafficAnomaly
expr: |
sum(rate(http_requests_total[5m])) by (service)
< sum(rate(http_requests_total[5m] offset 1h)) by (service) * 0.5
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.service }} traffic dropped 50% vs 1h ago"
---
FAQ
Q: What's the difference between Golden Signals, USE, and RED?
A: Golden Signals is the overarching framework. USE (Utilization, Saturation, Errors) applies to infrastructure. RED (Rate, Errors, Duration) applies to services. Use all three for full coverage.
Q: How do I avoid dashboard sprawl?
A: Start with one golden signals dashboard per service tier. Use Grafana variables for service/namespace selection instead of separate dashboards per service.
Q: What scrape interval should I use?
A: 15s for services, 30s for infrastructure. Match alerting for duration to at least 2x your scrape interval.
Q: Should I alert on all four signals?
A: Alert on Errors and Latency primarily. Use Traffic for anomaly detection. Saturation alerts prevent future problems. Not every signal needs a paging alert.
Q: How do I handle histogram bucket configuration?
A: Define buckets based on SLO. If SLO is p99 < 500ms, use [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]. Too few lose precision; too many waste cardinality.
---