Elasticsearch and OpenSearch have a health endpoint that everyone knows and almost nobody monitors correctly. GET /_cluster/health returns a status of green, yellow, or red in the body, and 200 OK in the HTTP status every time, including when the cluster is red and half your indices are unavailable. An uptime monitor pointed at it with the default settings is a monitor on whether the HTTP port is open. Yellow, meanwhile, is the state where everything still works and one more node failure loses data, and it can persist for weeks without anyone noticing because nothing is broken yet.
There is a one-parameter fix that most people have never used for monitoring, and it is the centre of this guide. The rest is the supporting cast: a least-privilege credential in a custom header, keyword checks for the numbers that explain why the cluster is yellow, a Kibana or OpenSearch Dashboards check, a search canary, and heartbeats for snapshots. Status checks and custom headers are on every CronAlert plan; keyword assertions and heartbeats are Pro ($5/month) and are marked. The examples use es.example.com:9200 behind TLS.
The trick: wait_for_status turns colour into a status code
The cluster health API accepts wait_for_status (green, yellow, or red) and timeout (default 30 seconds). It waits until the cluster reaches the requested status or better, or until the timeout expires. When it times out, the response body has "timed_out": true, and in both Elasticsearch's and OpenSearch's source the response's REST status is REQUEST_TIMEOUT when timed out and OK otherwise. In other words:
GET /_cluster/health?wait_for_status=green&timeout=5s
→ cluster green within 5s: HTTP 200, "status":"green", "timed_out":false
→ cluster yellow or red: HTTP 408, "status":"yellow", "timed_out":true That makes cluster colour visible to a monitor that only looks at status codes, which means the free plan can alert on a yellow cluster. Pick the colour that is normal for you: green for a multi-node cluster with replicas, yellow for a single-node cluster whose indices have number_of_replicas above zero (yellow is permanent there, and wait_for_status=yellow fails only on red). Keep the timeout short, 5 seconds, and set the monitor's own timeout above it, 10 seconds, so the monitor reads the 408 rather than giving up first. The documentation phrases the parameters as "wait (until the timeout expires) for" the condition, and the 408 mapping is in the source rather than the reference page, so test it once with curl -i on a cluster you have deliberately made yellow (set a replica count higher than your node count on a scratch index) before trusting it.
Step 1: a least-privilege credential
Cluster health needs authentication on any secured cluster, and the credential will live in a CronAlert monitor's custom headers, visible to your team. Make it one that can do nothing else.
- Elasticsearch: create a role with only the
monitorcluster privilege (the built-inmonitoring_userrole also works; the documentation describes it as granting access to the monitoring indices and the privileges necessary for reading basic cluster information). ThenPOST /_security/api_keywith that role; the response includes anencodedfield, the base64 ofid:api_key, which goes in the headerAuthorization: ApiKey <encoded>. Give the key an expiration and a calendar reminder; step 6 covers watching it. - OpenSearch: create a user and map it to the built-in
readall_and_monitorrole, which the documentation describes as read and search on all indexes plus cluster monitoring permissions such as viewing cluster health and statistics. Send basic auth:Authorization: Basic <base64 user:password>. If you want to avoid read access to data, create a role with justcluster_monitor-style permissions instead; the health endpoint needs only cluster monitoring.
The custom-headers guide covers how headers are stored and the basic-auth encoding. Do not use the elastic or admin superuser; a monitor is not where that password belongs.
Step 2: the cluster colour monitor
- Add an HTTP monitor on
https://es.example.com:9200/_cluster/health?wait_for_status=green&timeout=5s. - Custom headers:
{"Authorization": "ApiKey <encoded>"}(or the Basic header for OpenSearch). - Expected status
200, monitor timeout10seconds.
Green cluster, green monitor. Yellow or red cluster, 408, red monitor, and the check result's body tells you which ("status":"yellow"), along with number_of_nodes, unassigned_shards, relocating_shards, and number_of_pending_tasks. A connection error or 5xx means the node answering is down or the proxy in front is. If you point the monitor at a single coordinating node rather than a load balancer, you are also monitoring that node; for the cluster view, use the balanced address and add per-node monitors with wait_for_status omitted (plain 200) if you want to know about individual node loss separately.
Intervals are 3 minutes on Free and 1 minute on Pro. Recovery from yellow after a node restart can take minutes on large indices, so for clusters with routine restarts consider a maintenance window around deploys rather than loosening the check.
Step 3: the numbers behind the colour (Pro)
Green with two nodes where there should be three is a cluster that will go yellow the moment anything moves. Keyword assertions on the same endpoint, without wait_for_status so it always answers 200, catch the shape of the cluster rather than its colour:
| Monitor | Assertion | Catches |
|---|---|---|
| node count | contains "number_of_nodes":3 | a node that left the cluster while shards rebalanced and colour stayed green |
| no unassigned shards | contains "unassigned_shards":0 | the same thing yellow catches, but with the count in the alert |
| pending tasks | regex_not_match "number_of_pending_tasks":[1-9]\d{2,} | a master that is falling behind (hundreds of queued tasks) |
| OpenSearch leader | contains "discovered_cluster_manager":true | a cluster that has lost its manager node |
Each monitor carries one assertion, so these are separate monitors; two or three is usually enough. Keyword monitoring covers the match types, including regex.
Step 4: Kibana or OpenSearch Dashboards
Kibana's GET /api/status on port 5601 returns JSON with status.overall.level (available, degraded, unavailable, or critical), per-service levels for elasticsearch and savedObjects under status.core, and per-plugin levels. Unauthenticated callers, and callers without the monitor privilege, get a redacted response that still includes the overall level, so this one needs no credential. Add a monitor on https://kibana.example.com/api/status, expected 200, and on Pro assert contains "level":"available"; the documentation notes the response describes only the Kibana instance that answered, so behind a balancer add one per instance if you run several. Kibana's status being degraded while Elasticsearch is green usually means Kibana cannot reach the cluster, which is a different outage from the cluster being unhealthy, and users experience it as "the dashboards are down."
OpenSearch Dashboards is a Kibana fork; test /api/status on your version with curl before relying on it, and fall back to a keyword check on the login page if the route is absent.
Step 5: a search canary (Pro)
Green says the shards are allocated. It does not say your application's queries work: a mapping change that broke a field type, an index alias pointing at an empty index after a failed reindex, an ingest pipeline that stopped so the newest data never arrives. Add a monitor that runs a real query with a search-only key (an API key whose role grants read on one index, or an OpenSearch user with readall):
GET https://es.example.com:9200/products/_search?q=sku:ANCHOR-001&size=1
Authorization: ApiKey <search-only encoded key>
assert contains: "sku":"ANCHOR-001" Index a document that exists only for monitoring and never changes, and assert on one of its fields rather than on hits.total, which is present even when the hit is wrong. For freshness, have your ingest job write a marker document and ping a heartbeat only after it can read it back, the same pattern as monitoring Meilisearch and Typesense, which this canary mirrors.
Step 6: snapshots and the key itself (Pro)
Snapshot lifecycle management runs backups on a schedule, and nothing tells you when it stops. GET /_slm/policy/<id> returns last_success and last_failure, each with a snapshot_name and time, plus next_execution_millis; the guide is careful to say that a successful start does not guarantee the snapshot completed, so also read GET /_slm/stats for total_snapshots_failed. Reading them needs the read_slm cluster privilege. Run it from cron rather than a monitor, because the comparison is relative to now:
# every 6 hours: the nightly SLM policy succeeded within the last 30 hours and has not failed since
0 */6 * * * P=$(curl -fsS --max-time 15 -H "Authorization: ApiKey $ES_SLM_KEY" https://es.example.com:9200/_slm/policy/nightly) \
&& S=$(echo "$P" | jq '.nightly.last_success.time // 0') \
&& F=$(echo "$P" | jq '.nightly.last_failure.time // 0') \
&& [ $(( $(date +%s)*1000 - S )) -lt $((30*3600*1000)) ] && [ "$F" -lt "$S" ] \
&& curl -fsS --max-time 10 --retry 3 https://cronalert.com/api/heartbeat/YOUR-TOKEN Heartbeat expected interval 6 hours, grace 1 hour. On OpenSearch, the Snapshot Management plugin's policy explain endpoint under _plugins/_sm/policies/ gives the equivalent last-run state. Monitoring backups covers the restore test, which for a search cluster can be a restore of one index into a scratch cluster.
Finally, the API keys themselves expire. Set an expiration when you create them, and have the same cron job call GET /_security/api_key?id=<id> and skip the ping when expiration is within a week, so a monitoring credential never silently dies.
What the 408 trick cannot tell you
- Why. 408 means "not green within 5 seconds". The body in the check result has the counts; the step 3 monitors turn the usual reasons into their own alerts.
- Disk watermarks before they bite. A node crossing the high watermark starts relocating shards (colour may stay green) and at the flood-stage watermark makes indices read-only (still green). Watch
relocating_shardsas a keyword check, and alert on disk from your host monitoring. - Query latency. HTTP monitors have no latency alerting beyond the timeout; the search canary's response time in the check history is a trend, not an alarm. Timeouts explains the limit.
- Data correctness. That is the canary's job, and it covers one document. Broader validation belongs in your pipeline tests.
Reading an alert
| Alert | Likely cause | First thing to check |
|---|---|---|
| Colour monitor: expected 200, got 408 | Cluster yellow or red | "status" and "unassigned_shards" in the body; GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason |
| Colour monitor: 401 or 403 | API key expired or role changed | GET /_security/api_key; the application's credentials are probably fine, yours is not |
| Colour monitor: connection error or 502 | Node or proxy down | The node; the balancer |
| Colour green, node-count keyword red | A node left; shards rebalanced | GET /_cat/nodes?v |
Kibana degraded, cluster green | Kibana cannot reach Elasticsearch, or a plugin is unhealthy | status.core.elasticsearch and status.plugins in the body |
| Search canary 404 or keyword missing | Index deleted, alias moved, document gone | GET /_cat/aliases; the reindex job |
| SLM heartbeat overdue | Snapshot repository unreachable, policy stopped, or key expiring | last_failure in the policy; repository verify |
Frequently asked questions
Will a health request every minute load the cluster?
No. Cluster health is served from the master's cluster state and is what load balancers and Kubernetes probes already poll. The wait_for_status form holds a request open for up to 5 seconds when the cluster is unhealthy, which is negligible.
My single-node cluster is always yellow. What do I monitor?
Either set number_of_replicas: 0 on your indices (there is nowhere to put replicas) so it goes green, or use wait_for_status=yellow, which fails only on red. The second hides nothing on a single node, because yellow is the best it can be.
Can I use this on Elastic Cloud or Amazon OpenSearch Service?
Yes for the cluster health, Kibana, and canary checks, with the hosted endpoint and a key or user you create there; Amazon OpenSearch Service with fine-grained access control accepts basic auth, and IAM-signed requests are out of reach for an HTTP monitor. Snapshots on managed services are the provider's job.
Which parts are free?
The colour monitor (408 trick) with its custom header, the Kibana status check as a status-only monitor, and certificate monitoring are free. The keyword assertions, the search canary, and the SLM heartbeat need Pro at $5/month.
Get started
One free monitor with one extra query parameter turns the cluster-health endpoint from a port check into a colour check. Create a free CronAlert account, create a monitor-only API key, and add it. Related reading: monitoring Meilisearch and Typesense, monitoring PostgreSQL replication lag, monitoring Redis, Kubernetes uptime monitoring for clusters on ECK or the OpenSearch operator, and why a monitor can say UP while the service is broken.