Why Is My ElasticSearch/OpenSearch Cluster Red or Yellow? — Complete Fix Guide

TL;DR — Quick Fix

Check cluster health and fix unassigned shards immediately:

# Check cluster health
curl -s localhost:9200/_cluster/health?pretty

# Find unassigned shards and the reason
curl -s localhost:9200/_cluster/allocation/explain?pretty

# Quick fix: reduce replicas for red indices (emergency)
curl -XPUT localhost:9200/my-index/_settings -H 'Content-Type: application/json' -d '{
  "index.number_of_replicas": 0
}'

# Force retry shard allocation
curl -XPOST localhost:9200/_cluster/reroute?retry_failed=true

# Check disk watermarks (most common cause)
curl -s localhost:9200/_cat/allocation?v&h=node,disk.percent,disk.used,disk.avail,shards

---

Architecture — Cluster Health Decision Tree

---

Step 1 — Diagnose Unassigned Shards

# List all unassigned shards with reason
curl -s localhost:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED

# Get detailed allocation explanation
curl -s localhost:9200/_cluster/allocation/explain?pretty -H 'Content-Type: application/json' -d '{
  "index": "my-index",
  "shard": 0,
  "primary": true
}'

Common unassigned reasons:

---

Step 2 — Fix Disk Watermark Issues

# Check current disk usage per node
curl -s localhost:9200/_cat/allocation?v

# View current watermark settings
curl -s localhost:9200/_cluster/settings?include_defaults=true | \
  jq '.defaults.cluster.routing.allocation.disk'

# Temporarily raise watermarks (emergency only)
curl -XPUT localhost:9200/_cluster/settings -H 'Content-Type: application/json' -d '{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "90%",
    "cluster.routing.allocation.disk.watermark.high": "95%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "97%"
  }
}'

# Delete old indices to free space
curl -XDELETE "localhost:9200/logs-2025.01.*"

# Force merge to reclaim space from deleted documents
curl -XPOST "localhost:9200/large-index/_forcemerge?max_num_segments=1"

---

Step 3 — Index Lifecycle Management (ILM)

Prevent disk issues permanently with ILM policies:

# Create ILM policy
curl -XPUT localhost:9200/_ilm/policy/logs-lifecycle -H 'Content-Type: application/json' -d '{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "30gb",
            "max_age": "1d"
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "3d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "allocate": {
            "number_of_replicas": 1,
            "require": { "data_tier": "warm" }
          },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "allocate": {
            "number_of_replicas": 0,
            "require": { "data_tier": "cold" }
          },
          "set_priority": { "priority": 0 }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": { "delete": {} }
      }
    }
  }
}'

# Apply ILM to index template
curl -XPUT localhost:9200/_index_template/logs-template -H 'Content-Type: application/json' -d '{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "index.lifecycle.name": "logs-lifecycle",
      "index.lifecycle.rollover_alias": "logs-write",
      "number_of_shards": 3,
      "number_of_replicas": 1
    }
  }
}'

---

Step 4 — Prevent Split-Brain

# elasticsearch.yml — production settings
cluster.name: production-cluster

# Minimum master-eligible nodes for quorum (n/2 + 1)
discovery.seed_hosts:
  - master-1:9300
  - master-2:9300
  - master-3:9300

cluster.initial_master_nodes:
  - master-1
  - master-2
  - master-3

# Dedicated master nodes (no data)
node.roles: [master]

# Network settings
network.host: 0.0.0.0
transport.port: 9300

# Recovery settings
gateway.recover_after_data_nodes: 2
gateway.expected_data_nodes: 3

---

Step 5 — Recovery Procedures

# Recover from snapshot (for corrupted primary shards)
curl -XPOST localhost:9200/_snapshot/my-backup/snapshot-2026-08-01/_restore \
  -H 'Content-Type: application/json' -d '{
  "indices": "corrupted-index",
  "rename_pattern": "(.+)",
  "rename_replacement": "restored-$1"
}'

# Allocate stale primary (last resort — may lose data)
curl -XPOST localhost:9200/_cluster/reroute -H 'Content-Type: application/json' -d '{
  "commands": [{
    "allocate_stale_primary": {
      "index": "my-index",
      "shard": 0,
      "node": "data-node-1",
      "accept_data_loss": true
    }
  }]
}'

---

Node Right-Sizing Guide

Rule of thumb: Each shard should be 20-50GB. Keep total shards per node under 600.

---

Frequently Asked Questions

Why is my cluster yellow after adding a new index?

Yellow means replica shards are unassigned. If you have a single-node cluster, replicas can never be assigned (a replica cannot live on the same node as its primary). Set number_of_replicas: 0 for dev, or add more data nodes for production.

How do I know if my cluster is over-sharded?

Check _cat/shards | wc -l. If you have thousands of tiny shards (< 1GB each), you are over-sharded. Consolidate by reindexing with fewer shards, or use ILM shrink actions. Each shard consumes ~50MB heap, so 10K shards equals 500MB heap just for metadata.

Should I use hot-warm-cold architecture?

Yes, for any cluster handling time-series data (logs, metrics). Hot nodes use SSDs for recent data, warm nodes use HDDs for older data, and cold nodes use the cheapest storage. This typically reduces storage costs by 60-70% while keeping recent data fast.

How do I prevent a single large index from turning the cluster red?

Use ILM rollover to split large indices into time-based chunks. Set max_primary_shard_size: 30gb to auto-rollover. This limits blast radius — if one rolled-over index corrupts, only that time window is affected.

What is the fastest way to recover from a red cluster?

First, check if the missing node can be restarted. If yes, restart it and shards will auto-recover. If the node is gone permanently, use allocate_stale_primary with accept_data_loss: true as a last resort, or restore from snapshot.

---

Related Resources

• Prometheus High Memory OOM Fix
• Monitoring and Alerting for Production Systems
• Datadog vs New Relic vs Prometheus Comparison
• Linux Performance Troubleshooting
• DevOps Infrastructure Sizing Guide