A Consul or Nomad cluster is a small Raft group of servers plus a fleet of agents, and it degrades politely. Lose one server of three and everything keeps working; the only change is that the next failure takes the whole cluster with it. Lose the leader and the servers keep answering HTTP while writes stall and schedulers stop placing. Lose the last eligible Nomad client and the servers are perfectly healthy, with a queue of jobs that will never run. None of these show up on a monitor that checks whether port 8500 or 4646 answers.
Both products ship endpoints designed for external monitoring, including one with the rare property of changing its HTTP status code when the cluster is unhealthy. This guide uses them, plus a scheduled job that proves the scheduler works, to monitor either product from outside. Exact expected status codes, custom headers, and certificate monitoring are on every CronAlert plan; keyword assertions and heartbeats are Pro ($5/month) and are marked. Consul's HTTP API is on port 8500 by default (8501 for HTTPS), Nomad's on 4646; the examples use consul.example.com and nomad.example.com as load-balanced server addresses with TLS.
What each exposes
| Endpoint | Consul | Nomad | ACL | Monitor use |
|---|---|---|---|---|
/v1/status/leader | "10.1.10.12:8300" | "10.0.0.5:4647" | none | keyword: a leader exists |
/v1/status/peers | array of server addresses | same | none | keyword: expected peer count |
/v1/agent/health | — | {"client":{"ok":true},"server":{"ok":true}}; 500 when unhealthy | none | status: 200 |
/v1/operator/autopilot/health | Healthy, FailureTolerance, Servers[] | same | operator:read | status: 200 healthy, 429 unhealthy |
/v1/health/service/:name?passing=true | passing instances only | — | node:read, service:read | keyword: at least one instance |
/v1/health/state/critical | all critical checks | — | same | keyword: body is [] |
/v1/nodes | — | Status, SchedulingEligibility, Drain | node:read | keyword: a ready, eligible client |
/v1/snapshot | gzipped tar; management token | operator snapshot | management | heartbeat on the backup job |
The two status endpoints and Nomad's agent health need no token, which makes them the free-plan starting point. Everything else goes in a monitor's custom headers as X-Consul-Token or X-Nomad-Token with a policy that grants only what that monitor reads.
Step 1: is there a leader?
Add an HTTP monitor on https://consul.example.com:8501/v1/status/leader (and the Nomad equivalent on 4646), expected status 200, timeout 10 seconds. A 200 proves a server answered. On Pro, add the assertion that makes it meaningful: regex_match "\d+\.\d+\.\d+\.\d+:\d+", which fails when the body is an empty string, the shape of a cluster with no elected leader. Without a leader, Consul cannot commit writes and Nomad cannot schedule, while every node keeps answering reads, so this is the first check that tells you about the failure users will feel. /v1/status/peers with contains for each expected server address, or a regex counting three addresses, catches a server that has dropped out of the Raft configuration.
Point these at the load-balanced server address so any server can answer. Per-server monitors on the same path tell you which server is down; use plain 200 for those.
Step 2: autopilot health as a status code
This is the endpoint built for the job. GET /v1/operator/autopilot/health exists on both products with the same contract, quoted from the documentation: if Healthy is true, a status of 200 is returned; if Healthy is false, a status of 429 is returned. The body has FailureTolerance, the number of additional servers you can lose without losing quorum, and a Servers array with each server's Healthy, Leader, Voter, LastContact, and StableSince.
- Create an ACL token with a policy granting only
operator = "read". - Add a monitor on
https://consul.example.com:8501/v1/operator/autopilot/healthwith custom header{"X-Consul-Token": "<token>"}, expected status 200. Same for Nomad withX-Nomad-Token. - On Pro, add a second monitor on the same URL asserting contains
"FailureTolerance":1(for a three-server cluster), so you learn about the first server loss even whileHealthyis still true.
A 429 is the alert that a server is unhealthy, with the body naming which one and why (LastContact growing, SerfStatus not alive). This is the check that turns "we lost a server last Tuesday and found out during the outage on Friday" into an alert on Tuesday.
Step 3: Nomad agent health and clients
Nomad's GET /v1/agent/health needs no token and answers with a client and a server object, each with ok and message, and the documentation says a 500 is returned along with an error message when the agent is unhealthy. Monitor it on each server address (expected 200) for a token-free liveness check that goes deeper than the port. For the client fleet, GET /v1/nodes (node:read) lists every client with Status, SchedulingEligibility, and Drain; on Pro, a monitor asserting contains "SchedulingEligibility":"eligible" fails when every client is ineligible or gone, which is the state where the servers are healthy and nothing can run. The step 4 heartbeat is the stronger version of this check.
Step 4: a periodic job that pings a heartbeat (Pro)
Every endpoint above describes the cluster. Only a job proves the cluster can do its work. Nomad's periodic block schedules a batch job on a cron expression (six fields with seconds first; crons is the current list form, cron the deprecated single form) and prohibit_overlap keeps runs from stacking:
job "heartbeat" {
type = "batch"
datacenters = ["dc1"]
periodic {
crons = ["0 */15 * * * *"] # every 15 minutes, seconds field first
prohibit_overlap = true
}
group "ping" {
task "curl" {
driver = "docker"
config {
image = "curlimages/curl"
args = ["-fsS", "--max-time", "10", "--retry", "3",
"https://cronalert.com/api/heartbeat/YOUR-TOKEN"]
}
resources { cpu = 50, memory = 32 }
}
}
} Create a heartbeat monitor with an expected interval of 15 minutes and a grace of 15 minutes. The ping arrives only if the servers have a leader, the scheduler placed the allocation, a client was ready and eligible, and the Docker driver pulled and ran the image. If you run several node pools or datacenters, give each its own periodic job with a constraint so each pool has its own heartbeat. The same idea works for Consul indirectly: a service with a Consul health check that itself pings a heartbeat proves the agent is running checks. Cron job heartbeats and grace periods cover the mechanics; the Kubernetes CronJob guide is the same pattern on the other scheduler.
Step 5: Consul service health (Pro)
Consul's point is that other systems ask it which instances of a service are healthy, through DNS or the API. GET /v1/health/service/api?passing=true returns only instances whose checks all pass, as an array of Node, Service, and Checks; it needs node:read and service:read. A monitor with contains "Service":"api" (the service name field inside each entry) fails when the array is empty, which is exactly when Consul DNS stops returning the service and dependents start failing. One monitor per service you care about, and a last one on GET /v1/health/state/critical asserting contains [] as the entire body (use regex_match ^\[\]$) to catch any critical check anywhere. Note the response header X-Consul-Results-Filtered-By-ACLs: true: if your token cannot see a service, the array is empty for the wrong reason, so give the monitoring policy read on the services it checks.
Step 6: snapshots (Pro)
Consul's GET /v1/snapshot returns the Raft state as a gzipped tar archive (application/x-gzip) and requires a management token; the stale parameter lets any follower answer, and the matching CLI is consul snapshot save. Nomad has the equivalent under its operator commands. Automated snapshot agents are enterprise features on both, so on open-source clusters the backup is cron:
# /etc/cron.daily/consul-snapshot
set -euo pipefail
DEST=/srv/backups/consul/$(date +%F).snap
curl -fsS --max-time 120 -H "X-Consul-Token: $CONSUL_SNAPSHOT_TOKEN" \
"https://consul.example.com:8501/v1/snapshot?stale" -o "$DEST"
[ -s "$DEST" ] && tar -tzf "$DEST" >/dev/null # non-empty and a valid archive
# ship it off the host ...
curl -fsS --max-time 10 --retry 3 https://cronalert.com/api/heartbeat/YOUR-TOKEN Heartbeat expected interval 1 day, grace a few hours. The management token used here is the most powerful credential in the cluster; it lives on the cron host with tight permissions and nowhere near a monitor. Monitoring backups covers restore testing, which for Consul is a restore into a scratch cluster with consul snapshot restore.
Certificates and private clusters
Every agent and every client validates the server certificate, and an expired one is a cluster-wide outage. SSL monitoring is included with every HTTP monitor on every plan, so the step 1 and 2 monitors already watch the certificates on the server addresses. Clusters using an internal CA will fail CronAlert's public-root validation; for those, run the same curl checks with --cacert from a host inside the network and report via heartbeat, as in monitoring internal tools. Many Consul and Nomad clusters are entirely private, and the step 4 periodic job is then the single most valuable monitor, because it needs no inbound access at all.
Reading an alert
| Alert | Likely cause | First thing to check |
|---|---|---|
| Leader monitor: 200 but regex fails (empty body) | No elected leader: lost quorum or election in progress | consul operator raft list-peers / nomad operator raft list-peers |
| Autopilot: expected 200, got 429 | A server is unhealthy | Servers[] in the body: Healthy, LastContact, SerfStatus |
Autopilot 200, FailureTolerance keyword red | One server already lost; still healthy | Replace the server before the next failure |
| Autopilot 403 | Token revoked or policy changed | The monitoring token's policy |
Nomad /v1/agent/health 500 | Agent unhealthy; message says why | Body; agent logs |
| Periodic heartbeat overdue, servers green | No eligible client, driver failure, image pull failure | nomad job status heartbeat; the last allocation's events |
| Service health keyword red | Zero passing instances of that service | GET /v1/health/checks/<service> for the failing check output |
| Snapshot heartbeat overdue | Cron, token, disk | Run the script by hand |
Frequently asked questions
Should I monitor every server individually?
Monitor the balanced address for cluster state and autopilot, and add one plain-200 monitor per server on /v1/status/leader (or Nomad's /v1/agent/health) for host-level visibility. Autopilot's body already names unhealthy servers, so the per-server monitors are mainly for catching a host that is down while the cluster is still healthy.
Why not monitor the Consul DNS interface?
CronAlert's checks are HTTP and TCP, not DNS queries against an arbitrary resolver, and Consul's DNS port (8600 by default) is rarely public. The passing=true service health monitor is the HTTP view of the same answer DNS would give.
Does the 15-minute periodic job cost anything?
One tiny container for under a second, four times an hour. Set prohibit_overlap so a stuck run does not pile up, and prune old periodic allocations with Nomad's garbage collection settings if the history bothers you.
Which parts are free?
The leader and peers monitors as status checks, Nomad's agent health, the autopilot monitor with its 200/429 contract and a custom header, and certificate monitoring are free. The keyword assertions (leader regex, failure tolerance, service health, node eligibility) and the heartbeats (periodic job, snapshot) need Pro at $5/month.
Get started
Two free monitors, one on the leader endpoint and one on autopilot health expecting 200, cover the two failures that make a Raft cluster look fine until it is not. Create a free CronAlert account, create an operator:read token, and add them; the periodic heartbeat job comes next. Related reading: monitoring HashiCorp Vault and OpenBao, whose health endpoint plays the same status-code game; Kubernetes uptime monitoring; microservices monitoring for the services Consul fronts; monitoring Docker self-hosted services; and custom headers and exact status codes.