Prometheus High Memory Usage & OOM Crashes — Fix High-Cardinality Metrics

TL;DR — Quick Fix

# 1. Check current memory usage
curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data.headStats'

# 2. Find the top cardinality offenders
curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName[:10]'

# 3. Drop high-cardinality labels immediately via relabeling
# Add to prometheus.yml and reload

---

Why Prometheus Crashes With OOM

Prometheus stores every unique combination of metric name + label values as a separate time series in memory. This is called cardinality. When cardinality explodes (millions of unique series), Prometheus runs out of RAM and gets OOMKilled.

The formula:

Memory ≈ active_series x 2-3 KB (head block)

So 5 million active series needs approximately 10-15 GB of RAM just for the head block.

Step 1: Find Your Cardinality Offenders

Using Prometheus TSDB Status API

# Top metrics by series count
curl -s http://localhost:9090/api/v1/status/tsdb | \
  jq '.data.seriesCountByMetricName | sort_by(-.value) | .[0:10]'

# Top labels by value count (high-cardinality labels)
curl -s http://localhost:9090/api/v1/status/tsdb | \
  jq '.data.labelValueCountByLabelName | sort_by(-.value) | .[0:10]'

# Total head series count
curl -s http://localhost:9090/api/v1/status/tsdb | \
  jq '.data.headStats.numSeries'

Using PromQL Queries

# Count series per metric name
count({__name__=~".+"}) by (__name__)

# Find metrics with exploding cardinality
topk(10, count by (__name__)({__name__=~".+"}))

# Check series created in last hour (churn detection)
sum(increase(prometheus_tsdb_head_series_created_total[1h]))

Step 2: Common High-Cardinality Culprits

Step 3: Fix with Metric Relabeling

Add metric_relabel_configs to your prometheus.yml to drop or modify problematic metrics at ingestion time:

Drop Entire High-Cardinality Metrics

scrape_configs:
  - job_name: 'my-app'
    metric_relabel_configs:
      # Drop metrics you don't need
      - source_labels: [__name__]
        regex: 'go_gc_.|go_memstats_.'
        action: drop

      # Drop the high-cardinality 'path' label from HTTP metrics
      - source_labels: [__name__]
        regex: 'http_request_.*'
        action: replace
        target_label: path
        replacement: ''

Normalize URL Paths

metric_relabel_configs:
  # Replace dynamic path segments: /users/12345 -> /users/:id
  - source_labels: [path]
    regex: '/users/[0-9]+'
    target_label: path
    replacement: '/users/:id'

  - source_labels: [path]
    regex: '/orders/[a-f0-9-]+'
    target_label: path
    replacement: '/orders/:id'

Drop Unused Labels

metric_relabel_configs:
  # Remove noisy labels that add cardinality without value
  - regex: 'instance|pod_template_hash|controller_revision_hash'
    action: labeldrop

Step 4: Use Recording Rules for Expensive Queries

Instead of running expensive high-cardinality queries in dashboards, pre-aggregate with recording rules:

# prometheus-rules.yml
groups:
  - name: aggregated_metrics
    interval: 30s
    rules:
      # Pre-aggregate request rates by service (not by pod)
      - record: service:http_requests_total:rate5m
        expr: sum(rate(http_requests_total[5m])) by (service, method, status)

      # Pre-aggregate latency percentiles
      - record: service:http_request_duration_seconds:p99
        expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (service, le))

      # Pre-aggregate error rates
      - record: service:http_error_rate:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
          /
          sum(rate(http_requests_total[5m])) by (service)

Then use service:http_requests_total:rate5m in your Grafana dashboards instead of the raw metric.

Step 5: Configure Prometheus Resource Limits

Set Proper Storage and Query Flags

args:
  - '--storage.tsdb.retention.time=15d'
  - '--storage.tsdb.retention.size=50GB'
  - '--storage.tsdb.wal-compression'
  - '--query.max-samples=50000000'
  - '--query.timeout=2m'

Kubernetes Resource Configuration

resources:
  requests:
    memory: "4Gi"
    cpu: "1000m"
  limits:
    memory: "8Gi"

Sizing rule of thumb:

Required RAM = (active_series x 3 KB) + (ingestion_rate x 2 hours x sample_size)

For 2 million active series: ~6-8 GB RAM minimum.

Step 6: When to Move Beyond Single Prometheus

If you have more than 10 million active series, a single Prometheus instance won't cut it. Consider:

Option A: Thanos (Federation + Long-Term Storage)

# Thanos Sidecar — ship blocks to object storage
containers:
  - name: prometheus
    args:
      - '--storage.tsdb.min-block-duration=2h'
      - '--storage.tsdb.max-block-duration=2h'
  - name: thanos-sidecar
    image: quay.io/thanos/thanos:v0.35.0
    args:
      - sidecar
      - '--tsdb.path=/data'
      - '--objstore.config-file=/etc/thanos/bucket.yml'

Option B: VictoriaMetrics (Drop-In Replacement)

containers:
  - name: victoriametrics
    image: victoriametrics/victoria-metrics:v1.101.0
    args:
      - '-retentionPeriod=30d'
      - '-dedup.minScrapeInterval=15s'
    resources:
      requests:
        memory: "2Gi"  # Handles same load as 8GB Prometheus

Option C: Prometheus Sharding

Split scrape targets across multiple Prometheus instances:

# Instance 1 — scrapes services A-M
scrape_configs:
  - job_name: 'services-a-m'
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace]
        regex: 'service-[a-m].*'
        action: keep

# Instance 2 — scrapes services N-Z
scrape_configs:
  - job_name: 'services-n-z'
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace]
        regex: 'service-[n-z].*'
        action: keep

Monitoring Prometheus Itself

Set up alerts for Prometheus before it crashes:

groups:
  - name: prometheus-self-monitoring
    rules:
      - alert: PrometheusHighMemory
        expr: process_resident_memory_bytes{job="prometheus"} / 1024^3 > 6
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Prometheus using {{ $value | printf \"%.1f\" }}GB RAM"

      - alert: PrometheusHighCardinality
        expr: prometheus_tsdb_head_series > 5000000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Prometheus head series count: {{ $value }}"

      - alert: PrometheusHighChurn
        expr: rate(prometheus_tsdb_head_series_created_total[1h]) > 1000
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "High series churn: {{ $value }} new series/sec"

---

Frequently Asked Questions

How much memory does Prometheus need per time series?

Approximately 2-3 KB per active time series for the head block (last 2 hours of data). So 1 million active series needs roughly 2-3 GB of RAM. This doesn't include query memory or WAL overhead — budget 30-50% extra for those.

What is high cardinality in Prometheus?

Cardinality is the total number of unique time series (unique combinations of metric name and label values). High cardinality means millions of unique series, typically caused by labels with unbounded values like user IDs, request paths, or trace IDs. Anything above 5 million active series on a single instance is considered high.

How do I find which metrics use the most memory?

Use the TSDB status API: curl http://localhost:9090/api/v1/status/tsdb. Look at seriesCountByMetricName for metrics with the most series, and labelValueCountByLabelName for labels with the most unique values. The combination of these two views reveals your cardinality hotspots.

Should I switch from Prometheus to VictoriaMetrics?

Consider VictoriaMetrics if you need long-term retention beyond 30 days, your cardinality exceeds 10 million series, you want lower memory footprint, or you need built-in downsampling. VictoriaMetrics is a drop-in compatible replacement that accepts Prometheus remote_write and supports PromQL queries.

How do recording rules reduce memory usage?

Recording rules don't reduce ingestion cardinality directly. They pre-compute expensive aggregations so your dashboards query the pre-aggregated metric (low cardinality) instead of the raw metric (high cardinality). This reduces query-time memory spikes. Combine recording rules with metric_relabel_configs to drop the raw high-cardinality metrics after aggregation.

---

Related Resources

• Datadog vs. New Relic vs. Prometheus Comparison — Choosing your monitoring stack
• Monitoring & Alerting for Production Systems — End-to-end observability setup
• Kubernetes Resource Limits & Requests — Sizing Prometheus pods correctly