Who Monitors Prometheus? Turn the Alertmanager Watchdog Into a Dead Man's Switch

Prometheus and Alertmanager are the thing most teams rely on to find out something is wrong, which creates a blind spot shaped exactly like themselves. If Prometheus is OOM-killed, if rule evaluation stalls, if Alertmanager's config reload fails and it keeps running on a stale route, if the node it lives on can't reach the internet anymore, then nothing fires. Not because everything is fine, but because the component that would have told you is the one that broke. Teams discover this the slow way: a quiet week that turns out to have been a dead Prometheus, not a healthy fleet.

The Prometheus community solved this years ago with an alert that is designed to fire forever. kube-prometheus-stack ships it as Watchdog, and the idea is simple: route the constant stream of Watchdog notifications to something outside the cluster that alerts when the stream stops. Most installs leave it routed to a null receiver, so the signal exists and is thrown away. This guide points it at a CronAlert heartbeat instead, then adds the external HTTP checks that cover what the heartbeat can't.

The Watchdog alert

If you run kube-prometheus-stack, the rule already exists; skip to the next section. For a plain Prometheus install, add it yourself:

# /etc/prometheus/rules/watchdog.yml — an alert that is always firing, on purpose
groups:
  - name: watchdog
    rules:
      - alert: Watchdog
        expr: vector(1)
        labels:
          severity: none
        annotations:
          summary: "Alerting pipeline heartbeat. Always firing; its absence is the alert."

vector(1) needs no scrape data to evaluate, so it fires as long as Prometheus is up and evaluating rules. That is the point: it isolates the pipeline from the targets. Every notification Alertmanager sends for it proves Prometheus evaluated the rule, delivered it to Alertmanager, Alertmanager matched a route, and the receiver's HTTP request got out.

Route it to a heartbeat

Create a heartbeat monitor in CronAlert. Set the expected interval to 1 minute and the grace period to about 5 minutes: Alertmanager's re-send timing isn't to the second, and a 5-minute grace absorbs that jitter while still alerting within a few minutes of a real stop. Copy the ping URL. Then add a receiver and a route in Alertmanager. With kube-prometheus-stack, it goes in the chart's values, replacing the default route that sends Watchdog to null:

# kube-prometheus-stack values.yaml (excerpt)
alertmanager:
  config:
    route:
      receiver: default
      routes:
        - receiver: cronalert-watchdog
          matchers:
            - alertname = "Watchdog"
          group_wait: 0s
          group_interval: 1m
          repeat_interval: 1m
    receivers:
      - name: default
        # your Slack / PagerDuty / email receiver config
      - name: cronalert-watchdog
        webhook_configs:
          - url: https://cronalert.com/api/heartbeat/<token>
            send_resolved: false

For a plain install, the same route and receivers blocks go in alertmanager.yml, and amtool check-config alertmanager.yml validates them before you reload. Older configs use match: { alertname: Watchdog } instead of matchers; both work.

Three things make this work. repeat_interval: 1m tells Alertmanager to re-notify every minute even though the alert hasn't changed, which turns a single long-running alert into a stream of pings. send_resolved: false matters because Watchdog never resolves, but if it ever did (Prometheus restarted and the alert briefly cleared), you don't want a "resolved" webhook counted as a ping. And the heartbeat endpoint accepts POST with any body: Alertmanager sends its JSON payload of alerts, CronAlert ignores the body and records the arrival. Nothing has to be parsed on either side.

Within a minute of reloading, the monitor shows its first ping. Now stop Alertmanager (kubectl scale statefulset alertmanager-… --replicas=0, or systemctl stop alertmanager), and about six minutes later CronAlert alerts you from outside the cluster, through whichever channels your team has. Scale it back up; the pings resume and the incident resolves itself. That one test is worth doing on purpose once, so the first time you see the alert you already know what it means.

What the heartbeat proves, and what it doesn't

A quiet Watchdog heartbeat means one of these broke: Prometheus is down or not evaluating rules, Prometheus can't deliver to Alertmanager (wrong address after a rename, network policy), Alertmanager is down or wedged, the route no longer matches (someone edited the config and the reload failed, or succeeded with a typo), or the cluster can't make outbound HTTPS requests. The last one is worth dwelling on: if Alertmanager can't reach cronalert.com, it very likely can't reach Slack or PagerDuty either, so the heartbeat tells you your real alerts are also not going out.

It does not prove scrapes work. vector(1) evaluates happily with every target down, so keep your up == 0 and TargetDown alerts; they are the ones that fire when a node exporter dies, and the heartbeat is what guarantees they can fire. It also doesn't prove a specific receiver works: a revoked Slack token or a wrong PagerDuty routing key fails only inside that receiver, while the webhook to CronAlert sails through. Test real receivers deliberately, say monthly, with amtool alert add test_receiver severity=critical and a matching route, the same way you'd test any alert channel. The heartbeat covers everything the receivers share; the test covers what they don't.

If you run Alertmanager in HA mode, the replicas gossip and deduplicate, so only one of them sends each notification. The heartbeat sees one ping per interval regardless of replica count, and it keeps going as long as any replica is healthy, which is what you want.

External checks on the stack itself

The heartbeat says the pipeline is alive. A few HTTP monitors, on the free plan, say which part isn't when it goes quiet, and they also catch the stack's own front door being down while the pipeline is fine:

• Prometheus serves /-/healthy (the process is up) and /-/ready (it has loaded config and is serving queries). If Prometheus is reachable from outside through an ingress, monitor /-/ready expecting 200. It usually sits behind basic auth; add an Authorization: Basic … custom header to the monitor rather than exposing it. If it isn't reachable at all, skip this one; the heartbeat covers it.
• Alertmanager serves the same two endpoints. /-/ready returns 200 once config is loaded, so a failed reload at startup shows here.
• Grafana answers /api/health without authentication: 200 with "database": "ok" when it can reach its own database. Monitor that, not the root URL, because / redirects to /login, CronAlert follows redirects, and the login page is a 200 that proves nothing about dashboards. On Pro, a keyword on ok guards against a reverse proxy's own 200 error page.

These are internal tools in the sense covered by monitoring internal tools: use an endpoint that answers meaningfully without a session, and authenticate with a header if you must.

Reading the signals together

Watchdog heartbeat	Prometheus /-/ready	Alertmanager /-/ready	Most likely
Silent	Down	Down	The node or namespace. Both processes unreachable; check the cluster, the ingress, or the host.
Silent	Down	Up	Prometheus: OOM-killed, crash-looping on a bad rule file, or stuck. Alertmanager is fine but has nothing to send.
Silent	Up	Down	Alertmanager: down, or never became ready after a config change.
Silent	Up	Up	Both up, pipeline broken between or after them: Prometheus can't reach Alertmanager, the Watchdog route was edited away, or outbound HTTPS is blocked. Check prometheus_notifications_errors_total and Alertmanager's logs.
Fine	Down	Up	Prometheus is evaluating rules but the ingress or auth in front of it broke. The pipeline works; the UI doesn't.
Fine	Up	Up	Healthy pipeline. If real alerts still aren't arriving, the problem is inside a specific receiver; run the amtool test.

Things that take the pipeline down quietly

• Memory. Prometheus's heap grows with series count, and a new high-cardinality label can push it past its limit weeks after the deploy that added it. The OOM kill is silent to Prometheus-based alerting by definition.
• Config edits. A route or receiver typo that fails validation leaves Alertmanager running on its previous config; one that passes validation but matches nothing sends alerts to the default receiver, or to null. The heartbeat catches the route change if it affects Watchdog; keep the Watchdog route at the top so edits below it don't swallow it.
• Renames and moves. Moving Alertmanager to another namespace or changing its Service name breaks Prometheus's alertmanagers target, and Prometheus logs the delivery failures to a log nobody reads.
• Egress policy. A new NetworkPolicy or egress firewall rule that cuts outbound HTTPS from the monitoring namespace stops every receiver at once. The heartbeat is the first and often only thing that notices.
• Upgrades. A chart upgrade restarts both components; put a maintenance window over planned ones, and let the grace period absorb the rest.

Frequently asked questions

Why not just HTTP-check Prometheus and Alertmanager?

Do that too, but a ready Prometheus with a broken route to Alertmanager, or an Alertmanager with no egress, passes both checks and delivers nothing. The heartbeat tests the path alerts actually take.

Can I use this with Thanos, Mimir, or Cortex?

Yes. Add the Watchdog rule wherever your rules are evaluated (the Thanos Ruler or Mimir ruler), and route it in whichever Alertmanager those send to. It then proves that pipeline instead.

What interval should the heartbeat have?

Match repeat_interval: 1 minute, with roughly 5 minutes of grace. Longer repeat intervals work, with correspondingly slower detection.

Does the Watchdog alert spam my other receivers?

Not if its route is matched before the default: a matched route with no continue: true stops there. That's also why it should sit first in the routes list.

Which plan?

The heartbeat is on Pro ($5/mo). The HTTP checks on the health endpoints, including basic-auth headers, are free.

Let the pipeline prove itself

A Watchdog alert routed to null is a smoke detector with the battery removed. Route it to a heartbeat and the whole chain, from rule evaluation to outbound HTTP, has to work once a minute or you hear about it from outside. Add free HTTP checks on the health endpoints to learn which link broke. Create a free account, add the health checks, and upgrade to Pro for the heartbeat before the next quiet week. Related reading: Kubernetes uptime monitoring, monitoring Kubernetes CronJobs with heartbeats, uptime monitoring vs APM vs logging for where external checks fit beside a metrics stack, and choosing between HTTP, TCP, and heartbeat monitors.