TL;DR — Quick Fix
If you're alerting on average latency, switch to percentiles immediately:
# p99 latency — what 1% of users actually experience
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
# Compare average vs p99 to see how much averages hide
# Average (misleading):
rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m])
# p99 (reality for 1% of users — usually 5-20x higher):
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
If your average is 50ms but p99 is 2000ms, 1 in 100 users waits 40x longer.
---
Why Average Latency Hides Problems
Averages mask the experience of your worst-affected users.
---
Percentile Math Explained
| Percentile | Meaning | Use Case |
|---|---|---|
| p50 (median) | 50% of requests faster | General baseline |
| p90 | 90% faster, 10% slower | Good for dashboards |
| p95 | 95% faster, 5% slower | Common SLO target |
| p99 | 99% faster, 1% slower | Critical services |
| p99.9 | 999/1000 faster | Payment/auth |
At 1000 req/s, p99 = 2s means 10 users every second wait 2+ seconds.
---
Prometheus histogram_quantile Setup
Instrumenting Your Application
# Python (prometheus_client)
from prometheus_client import Histogram
REQUEST_LATENCY = Histogram(
'http_request_duration_seconds',
'Request latency in seconds',
['method', 'endpoint', 'status'],
buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
)
@app.middleware("http")
async def track_latency(request, call_next):
start = time.time()
response = await call_next(request)
duration = time.time() - start
REQUEST_LATENCY.labels(
method=request.method,
endpoint=request.url.path,
status=response.status_code
).observe(duration)
return response
// TypeScript (prom-client)
import { Histogram } from 'prom-client';
const httpRequestDuration = new Histogram({
name: 'http_request_duration_seconds',
help: 'Request latency in seconds',
labelNames: ['method', 'route', 'status_code'],
buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
});
app.use((req, res, next) => {
const end = httpRequestDuration.startTimer();
res.on('finish', () => {
end({ method: req.method, route: req.route?.path || req.path, status_code: res.statusCode });
});
next();
});
---
Setting Up Histogram Buckets
# Choose buckets based on your SLO
# API service (fine-grained below 500ms)
buckets: [0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.15, 0.2, 0.3, 0.5, 0.75, 1.0, 2.5, 5.0]
# Batch processing (coarser, higher range)
buckets: [0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0, 60.0, 120.0, 300.0]
# General purpose (Prometheus defaults)
buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
Bucket Design Rules
---
Grafana Heatmap Visualization
# Grafana heatmap panel configuration
panels:
- title: "Request Latency Heatmap"
type: heatmap
targets:
- expr: sum(increase(http_request_duration_seconds_bucket{job="$service"}[1m])) by (le)
format: heatmap
legendFormat: "{{ le }}"
options:
calculate: false
yAxis:
unit: "s"
color:
mode: scheme
scheme: Spectral
# PromQL for comprehensive latency dashboard
# Percentiles over time
histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{service="api"}[5m])) by (le))
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service="api"}[5m])) by (le))
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service="api"}[5m])) by (le))
# Apdex score (satisfied < 0.5s, tolerating < 2s)
(
sum(rate(http_request_duration_seconds_bucket{le="0.5"}[5m]))
+ sum(rate(http_request_duration_seconds_bucket{le="2.0"}[5m]))
) / 2 / sum(rate(http_request_duration_seconds_count[5m]))
# Find slow endpoints
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, endpoint))
---
SLO Based on p99
# prometheus-rules/latency-slo.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: latency-slo-rules
spec:
groups:
- name: latency-slo
rules:
- record: slo:http_request_latency:success_ratio
expr: |
sum(rate(http_request_duration_seconds_bucket{le="0.5"}[5m])) by (service)
/ sum(rate(http_request_duration_seconds_count[5m])) by (service)
- record: slo:http_latency:error_budget_remaining
expr: |
1 - ((1 - slo:http_request_latency:success_ratio) / (1 - 0.999))
- alert: LatencySLOBudgetBurning
expr: slo:http_latency:error_budget_remaining < 0.5
for: 5m
labels:
severity: warning
annotations:
summary: "{{ $labels.service }} consumed 50%+ latency error budget"
- alert: P99LatencyHigh
expr: |
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)) > 0.5
for: 5m
labels:
severity: critical
annotations:
summary: "{{ $labels.service }} p99 exceeds 500ms SLO"
---
Real Examples: Average vs Percentile Divergence
# Scenario 1: DB connection pool exhaustion
# Average: 45ms (looks fine) | p99: 8500ms (pool queue timeout)
# Fix: Increase pool size from 10 to 50
# Scenario 2: Garbage collection pauses
# Average: 30ms (healthy) | p99: 1200ms (GC stop-the-world)
# Fix: Reduce heap size, tune GC parameters
# Scenario 3: Cold cache hits
# Average: 80ms (ok) | p99: 3000ms (cache miss → slow DB query)
# Fix: Increase cache TTL, pre-warm popular keys
# Find worst p99/avg ratio endpoints
# PromQL:
# histogram_quantile(0.99, sum(rate(bucket[5m])) by (le, endpoint))
# / (sum(rate(sum[5m])) by (endpoint) / sum(rate(count[5m])) by (endpoint))
---
FAQ
Q: Why not just use p99.9 or max latency?
A: p99.9 and max are too noisy — single outliers trigger alerts. p99 balances sensitivity with stability. Use p99 for alerting, p99.9 for dashboards, max for debugging.
Q: How does histogram_quantile work?
A: Prometheus histograms use cumulative counters per bucket. The function interpolates between boundaries to estimate percentiles. Accuracy depends on bucket placement near your SLO.
Q: What's the cardinality cost of histograms?
A: N buckets x L label combinations = total series. With 10 buckets and 200 label combos = 2000 series per service. Keep buckets under 15 and limit label cardinality.
Q: Should I use histograms or summaries?
A: Histograms for server-side (aggregatable across instances). Summaries only for client-side or single-instance exact percentiles. Summaries cannot be aggregated.
Q: How do I set an appropriate p99 SLO?
A: Measure baseline p99 for 2 weeks. Set SLO at ~2x baseline. For 200ms baseline, a 500ms SLO gives headroom while catching regressions.
---