Observability stacks have a particular blind spot: they are how you find out things are broken, so when they break, you find out nothing. Loki stops ingesting because its ingester cannot flush to a full S3 bucket, and the error that would tell you is a log line Loki did not store. Tempo's distributor goes down and traces vanish from every dashboard, which reads as "the app got faster". The Alloy agents on half the fleet are rejecting their config after a bad rollout, and the dashboards for those hosts show flat lines, which look like calm. Grafana itself goes down and the whole thing is invisible until someone opens a browser.

This guide monitors the stack from outside it, layer by layer: is Loki ready, is Tempo ready, are the agents healthy, does the tenant gateway route, is Grafana up, and the one question none of the health routes can answer, are logs actually arriving. Each layer is one CronAlert monitor. The HTTP checks run on the free plan; keyword assertions and the heartbeat are Pro.

Layer 1: Loki is ready

Every Loki process, whether a single binary, a simple-scalable read/write/backend trio, or a microservices deployment, serves a readiness route:

URL:              https://loki.example.com/ready
Method:           GET
Expected status:  200
Keyword (Pro):    contains "ready"

Loki answers 200 ready when the process's components are all up, and 503 otherwise, with a body that names the problem: an ingester still in its post-start wait, a ring not yet healthy, or, the one you care about, an ingester that cannot flush. A 503 for a minute after a restart is normal. A 503 that persists almost always means the storage backend (S3, GCS, MinIO, a disk) is full or unreachable, the flush queue is growing, and anything the ingester holds in memory is at risk. In the scalable deployment, monitor the write and read hostnames separately, since an ingester-side flush problem leaves reads green. If MinIO is the backend, monitor it directly too; that alert comes first.

Loki has no authentication of its own and expects a gateway or reverse proxy in front. Whatever that proxy requires, basic auth, a bearer token, or a Cloudflare Access service token, goes in the monitor's custom headers. Expose the health and read paths to the monitor's route; keep /loki/api/v1/push off it.

Layer 2: the tenant gateway routes

Readiness is per process. It does not prove that a request with your tenant's header gets through the gateway, hits a querier, and returns data. The cheapest query that does all of that is the label list:

URL:              https://loki.example.com/loki/api/v1/labels
Expected status:  200
Custom headers:   { "X-Scope-OrgID": "prod", "Authorization": "Basic <gateway credential>" }
Keyword (Pro):    contains "\"status\":\"success\""

In multi-tenant mode (auth_enabled: true) a missing or wrong tenant header fails the request, so this monitor also catches a gateway rewrite that stopped injecting the header. In single-tenant mode the header is ignored and the request uses the default tenant, which is fine. If the response is a 200 without success in the body, something other than Loki answered, usually a proxy's maintenance page.

Layer 3: Tempo is ready

Tempo serves the same readiness route as Loki, because they share the underlying framework:

URL:              https://tempo.example.com/ready
Expected status:  200

Tempo's silent failure is the distributor stopping while everything else runs: traces stop arriving, nothing errors loudly, and dashboards show fewer spans, which nobody alerts on. In a microservices deployment put the readiness monitor on the distributor's hostname specifically, and on the query-frontend's. Tempo's trace-by-ID and search APIs need a tenant header in multi-tenant mode in the same way as Loki, so the pattern from layer 2 applies if you want a query-path check.

Layer 4: the agents are healthy (Alloy and Promtail)

Loki and Tempo can be perfectly ready while nothing is sending to them. Grafana Alloy, the collector that replaced Promtail and Grafana Agent, has two routes that describe two different things:

URL:              http://alloy-01.example.com:12345/-/ready
Expected status:  200          (503 = initial config not loaded)

URL:              http://alloy-01.example.com:12345/-/healthy
Expected status:  200          (500 = a component is unhealthy, body says which)

/-/ready flips to 200 once the initial configuration loads and is the Kubernetes readiness probe. /-/healthy is the one an external monitor should watch: it returns 500 with a message when any component is unhealthy, for example a loki.write whose endpoint is rejecting pushes or a scrape target that is gone. Grafana's documentation says not to use it as a liveness probe because an unhealthy component does not need a restart, which is exactly why it belongs on a monitor rather than in a probe. Promtail, if you still run it, serves /ready the same way Loki does.

Agents are many and usually on private addresses, which public HTTP checks cannot reach, and a bare IP cannot be reached by HTTP checks from Cloudflare's network at all. Three options, in order of effort: monitor the two or three agents whose hosts matter most through a tunnel or a hostname; run the canary heartbeat in layer 6 from a representative host, which proves that host's agent end to end; or rely on Alloy's own Prometheus metrics in Grafana and give Alertmanager a dead man's switch so you learn when that stops. Most teams do the last two.

Layer 5: Grafana itself

Grafana exposes an unauthenticated health route that reports the database connection and version, and it is the check that tells you the dashboards are reachable at all:

URL:              https://grafana.example.com/api/health
Expected status:  200
Keyword (Pro):    contains "\"database\": \"ok\""

Grafana's database being ok is what makes the login and every saved dashboard work; the route returns a non-200 when it is not. Behind single sign-on, the login page redirects to the identity provider and a monitor on / measures that instead, so use /api/health, which does not go through the login flow. Monitoring the identity provider covers the case where Grafana is fine and nobody can sign in anyway.

Layer 6: logs are actually arriving (the canary line)

Everything above is a component describing itself. Loki can be ready, the gateway can route, the agents can be healthy, and logs can still not be arriving, because a tenant's ingestion rate limit is being hit, a distributor is rejecting out-of-order writes, a label on every line was renamed and the retention rule now deletes them, or the agent's positions file was reset and it is re-shipping a week of old logs while new ones wait. The only check that catches "logs are not arriving" is a log that you write and then try to read.

#!/bin/bash
# Push one line with a unique marker, read it back, ping only if found
LOKI=https://loki.example.com; TENANT=prod; AUTH="user:pass"
MARK="canary-$(date +%s)"; NOW=$(date +%s%N)
curl -fsS --max-time 10 -u "$AUTH" -H "X-Scope-OrgID: $TENANT" -H "Content-Type: application/json" \
  -X POST "$LOKI/loki/api/v1/push" \
  -d "{\"streams\":[{\"stream\":{\"job\":\"cronalert-canary\"},\"values\":[[\"$NOW\",\"$MARK\"]]}]}"
sleep 5
curl -fsS --max-time 15 -u "$AUTH" -H "X-Scope-OrgID: $TENANT" \
  "$LOKI/loki/api/v1/query_range" --data-urlencode "query={job=\"cronalert-canary\"} |= \"$MARK\"" \
  --data-urlencode "start=$(( $(date +%s) - 120 ))000000000" \
  | grep -q "$MARK" \
  && curl -fsS --max-time 10 --retry 3 https://cronalert.com/api/heartbeat/<loki-canary-token>

Run it every 5 minutes from inside the network, with the heartbeat's expected interval at 5 minutes and a grace of 5. If you would rather test the agent path than the push API, have the script append the marker line to a file that Alloy tails instead of pushing directly; the readback is the same, and a ping then proves Alloy shipped it. Grafana ships a loki-canary binary that does a continuous version of this with its own metrics; the script here is the heartbeat-shaped equivalent for teams who want the alert to come from outside the stack it is testing. The same shape works for Tempo with a trace instead of a line, if trace loss is something you would act on. Heartbeat monitoring covers interval and grace choices for scripts like this.

Reading the alerts

AlertWhat brokeFirst thing to check
Loki /ready 503 for more than a few minutesIngester cannot flush: storage full or unreachableThe 503 body; object storage health and free space
Loki /ready down entirelyProcess, host, or gatewayPods or containers; the gateway's logs
Ready green, labels check failingGateway, tenant header, or queriersGateway config; querier logs
Tempo /ready downDistributor or query-frontendWhichever hostname alerted
Alloy /-/healthy 500A component: write endpoint rejecting, target gone, bad configThe message in the body; alloy logs on that host
Alloy /-/ready 503Config failed to load after a rolloutThe rollout; config syntax
Grafana /api/health redGrafana or its databaseGrafana's database; disk
Canary heartbeat overdue, everything else greenIngestion rejected: rate limits, out-of-order, retention, agent stuckDistributor logs for the tenant; loki_discarded_samples_total

Alert channels in CronAlert are team-wide. If the platform team wants pages for the stack and application teams want a Slack notice, put these monitors in their own team; alert routing with teams explains the pattern.

Frequently asked questions

Isn't Prometheus already monitoring all of this?

Internally, yes, and in far more detail. The point of external checks is that they still work when the stack they describe does not. Keep the Prometheus alerts; add these so you hear about Prometheus and Loki being down from somewhere that is not Prometheus and Loki. Uptime monitoring vs APM vs logging covers where each sits.

We use Grafana Cloud for Loki and Tempo. What applies?

Layers 4 and 6. Grafana runs the backends; you run the agents and you own whether logs leave your network. Alloy's health routes and the canary heartbeat are exactly as useful against a hosted Loki.

Can CronAlert run the LogQL query itself instead of a script?

A GET on /loki/api/v1/query_range with the query in the URL and the tenant header works as an HTTP monitor, with a keyword on "status":"success", but it only proves the read path. Writing the canary line needs a POST body, which HTTP monitors do not send, so the write-then-read proof stays a heartbeat script.

How often should the canary run?

Every 5 minutes is enough for most teams; the heartbeat tells you within about 10 minutes that ingestion stopped. Teams with tight log-based alerting SLAs run it every minute on Pro.

Which plan do I need?

The readiness, labels, Tempo, Alloy, and Grafana monitors are HTTP checks with custom headers and an expected status, on every plan at 3-minute intervals on Free and 1-minute on Pro. Keyword assertions and the canary heartbeat are Pro ($5/mo, or $4/mo billed annually).

Monitor the stack from outside it

A logging stack that is down reports nothing, and nothing looks a lot like quiet. Readiness on Loki and Tempo, health on the agents, a tenant-aware query, Grafana's own health route, and one log line written and read back every five minutes turn that quiet into an alert with a layer attached. Create a free account, add the readiness and Grafana checks, and upgrade to Pro for the canary heartbeat before the next full bucket. Related reading: a dead man's switch for Prometheus and Alertmanager, Kubernetes uptime monitoring, microservices monitoring, monitoring Docker and self-hosted apps, and choosing between HTTP, TCP, and heartbeat monitors.