A container registry is unusual among the things you run: when it goes down, nothing breaks. Every container already running keeps running, because its image is already on the node. The outage waits. It waits for the next deploy, the next node that reboots and has to re-pull, the next autoscaling event at 2 a.m., and then everything that needed an image fails at once with ImagePullBackOff, x509: certificate has expired, or unauthorized: authentication required, and the on-call engineer spends the first twenty minutes convinced the problem is Kubernetes.
Registries also fail in layers. The HTTP front door can be up while the storage backend behind it is unreachable, so every pull of a blob 500s. Harbor's core can be up while its token service, database, or Redis is not, so every pull is a 401 that looks like a credentials problem. The certificate can expire, which no registry component reports and every node refuses. And the replication to the DR site, the garbage collection that keeps the disk from filling, and the vulnerability scans can all stop running without a single error reaching a human. This guide puts one CronAlert monitor on each layer, for Harbor and for the plain distribution registry. The HTTP checks and SSL expiry are on the free plan; keyword assertions and heartbeats need Pro.
Layer 1: the registry API answers (the expected-401 check)
Every registry that speaks the Docker Registry HTTP API V2, which is all of them, serves GET /v2/. The specification defines two healthy answers: 200 with an empty JSON object when the registry needs no authentication, and 401 with a WWW-Authenticate header when it does. Harbor, GitLab, Gitea, and any distribution registry with token auth return the 401. That makes the first monitor an inverse one:
URL: https://registry.example.com/v2/
Method: GET
Expected status: 401 (200 if your registry has no auth)
Keyword (Pro): contains "UNAUTHORIZED" The 401 is the point. It is the registry, not a reverse proxy or a load balancer, saying "I am here and I want a token", and the keyword (the error code in the JSON body) confirms the registry produced it rather than an auth proxy in front. A 502 or timeout is the registry or the proxy gone. A 404 means something else answered on that hostname. A 503 from a distribution registry has a specific meaning worth knowing: with the storage-driver health check enabled, distribution starts returning 503 to every request after three consecutive failures to reach its backend, so an S3 or MinIO outage behind the registry surfaces here before a single pull is attempted. The custom headers guide explains why a non-200 expected status is a first-class monitor rather than a workaround; the GitLab and Gitea guides use the same check on their built-in registries.
Layer 2: Harbor's own health (every component)
Harbor is a dozen containers with one hostname. Its core can serve the portal while the job service, database, Redis, registry, or Trivy scanner is down, and each of those produces a different user-visible failure. Harbor exposes a per-component health endpoint that needs no authentication on a default install:
URL: https://registry.example.com/api/v2.0/health
Expected status: 200
Keyword (Pro): not_contains "unhealthy" The body is JSON with an overall status and a components list, each entry a name, a status, and an error message when it is failing. The overall status is healthy only when every component is. Assert not_contains "unhealthy" and not contains "healthy": the second is satisfied by the word unhealthy, a trap that Airflow's health endpoint shares. If you only have the free plan, the status code still tells you Harbor's API is serving, and the component detail is one click away when something else goes wrong.
/api/v2.0/ping returns Pong and only proves the API process is up; it is a lighter check if you want one, but with intervals set by plan rather than per monitor, one monitor on /health does the job. Treat a health alert with the portal still loading as the early one: Redis down means the job queue is dead and nobody has noticed yet; database down means every pull fails at the token step; registry unhealthy means the storage backend is the problem.
Layer 3: a token is granted (the pull path, up to the blobs)
The thing every docker pull does after the 401 is ask the token service for a bearer token scoped to the repository. In Harbor, the token service lives at /service/token, and for a public project it grants an anonymous pull token without credentials. Create a small public project for the purpose (canary), push one tiny image into it, and monitor the grant:
URL: https://registry.example.com/service/token?service=harbor-registry&scope=repository:canary/ping:pull
Expected status: 200
Keyword (Pro): contains "\"token\":" This exercises the path the /v2/ check does not: core is up, it can talk to the database, it found the project, it evaluated the project's public flag, and it signed a token. When the database is down this returns a 500; when someone flips the project to private it returns a 401; both are things you want to know, and both leave /v2/ perfectly green. One monitor gets one assertion, so if you also want to prove the manifest is served, do it from the inside in layer 6 rather than trying to pack it in here.
On a plain distribution registry with no auth, skip the token and monitor the manifest directly. Send the Accept header the registry expects, or it may answer with a schema you did not intend:
URL: https://registry.example.com/v2/canary/ping/manifests/latest
Expected status: 200
Custom headers: { "Accept": "application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json" } A registry fronted by a token server (htpasswd or an external auth) answers 401 here for anonymous requests, so the /v2/ check is as far as an unauthenticated monitor can go, and the in-cluster pull canary covers the rest.
Layer 4: the certificate
Container runtimes are strict about TLS, and a registry certificate that expired at midnight produces x509: certificate has expired or is not yet valid on every node that needs to pull, with nothing in the registry's own logs. Registries also tend to carry certificates renewed by a different mechanism than the web tier: a cert-manager issuer, a file in Harbor's harbor.yml, or a copy made by hand two years ago. Enable SSL monitoring on the registry hostname and on any separate notary or token-service hostname. It is on every plan and alerts days ahead. SSL certificate monitoring covers the thresholds, and monitoring the renewal job catches the case where the cert renewed on disk and the registry was never restarted to load it.
Layer 5: the jobs ran (replication, garbage collection, scans)
Harbor's scheduled work fails by not happening. Replication to the secondary site stops, and you learn about it during the failover. Garbage collection stops, and you learn about it when the volume fills. Scheduled scans stop, and you learn about it in an audit. None of these produce an alert, because an alert is something a running job sends.
Harbor's webhook notifications (per project, under Webhooks) can POST a JSON payload to a URL when events happen, including replication status changes and scan completions. CronAlert heartbeat URLs accept POST and ignore the body, so a webhook pointed at a heartbeat records every completed run:
Harbor project webhook
Endpoint URL: https://cronalert.com/api/heartbeat/<replication-token>
Event types: Replication status changed
CronAlert heartbeat monitor
Expected interval: matches the replication schedule (e.g. 1 hour)
Grace period: the longest normal run, plus slack Be honest with yourself about what this proves: the status-changed event fires for failed runs as well, so the heartbeat means "replication ran", not "replication succeeded". For a success-only signal, or for garbage collection, which has no webhook event, run the job from a cron on a host you control and ping only on exit code 0. A skopeo sync or crane copy of the critical repositories to the DR registry works as a replication you fully control:
0 * * * * skopeo sync --src docker --dest docker --all \
registry.example.com/prod dr-registry.example.com/prod \
&& curl -fsS --retry 3 https://cronalert.com/api/heartbeat/<sync-token> For garbage collection, set the Harbor GC schedule and separately a cron that calls GET /api/v2.0/system/gc with a robot account, checks the newest job's status is Success and its time is recent, and pings a heartbeat. It is a few lines of shell and the only way to know GC is still happening. Batch job monitoring covers interval and grace choices for jobs whose duration varies, and the grace-period guide the trade-off between catching lateness and false alarms.
Layer 6: the nodes can actually pull
Everything above is measured from the internet. Your nodes pull from inside the cluster, through their own DNS, their own CA trust store, their own pull secrets, and sometimes a pull-through cache. The check that fails for the same reasons real pulls fail is a real pull, run where real pulls run. A Kubernetes CronJob that pulls a tiny canary image with imagePullPolicy: Always and pings a heartbeat when it starts does exactly that:
apiVersion: batch/v1
kind: CronJob
metadata:
name: registry-pull-canary
spec:
schedule: "*/10 * * * *"
concurrencyPolicy: Forbid
jobTemplate:
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
imagePullSecrets:
- name: harbor-pull
containers:
- name: canary
image: registry.example.com/canary/ping:latest
imagePullPolicy: Always
command: ["sh", "-c", "wget -qO- https://cronalert.com/api/heartbeat/<pull-token>"] Set the heartbeat to a 10-minute interval with a 10-minute grace. If the pull fails, the container never starts and the ping never happens. Rotate the pull secret and forget to update it here, and you find out in ten minutes instead of at the next deploy. Point a second copy of this at the pull-through cache if you run one. The Kubernetes CronJob monitoring guide covers the details of CronJob heartbeats, and the CI/CD monitoring guide the equivalent for the build runners that push.
If you run proxy-cache projects that front Docker Hub or GHCR, their upstreams are dependencies too. https://registry-1.docker.io/v2/ and https://ghcr.io/v2/ both answer 401 anonymously, so the same expected-401 monitor tells you when a cache miss is about to fail for reasons that are not yours; third-party dependency monitoring covers how to alert on those without paging for them.
Reading the alerts
| Alert | What broke | First thing to check |
|---|---|---|
/v2/ down, everything down | Registry, proxy, or host | docker compose ps in the Harbor directory; proxy logs |
/v2/ 503 | Storage backend unreachable (distribution's health check tripped) | The S3 or MinIO behind it |
/v2/ fine, Harbor health red | One component: database, Redis, jobservice, trivy, registry | The unhealthy entry's error message |
| Token grant 500 | Core cannot reach the database | Database container, connections, disk |
| Token grant 401 | Canary project made private, or auth mode changed | Project settings |
| Pull canary overdue, everything else green | Pull secret, node DNS, node CA trust, or the pull-through cache | kubectl describe pod on the canary job's pod |
| Replication heartbeat overdue | Replication schedule not firing, or jobservice dead | Harbor Replications page; jobservice logs |
| SSL expiring | Renewal missed, or renewed and not reloaded | Certificate dates; docker compose restart proxy |
Alert channels in CronAlert are team-wide. If the platform team wants pages for the registry and application teams want Slack for their services, put the registry monitors in their own team; alert routing with teams explains the pattern.
Frequently asked questions
Is a 401 from /v2/ really "up"?
Yes, by the specification. Unauthenticated clients are supposed to receive a 401 with a challenge, obtain a token, and retry. The registry answering 401 is doing its job. It becomes a problem when a monitor expects 200 and treats every correct answer as an outage, which is why you set the expected status to 401 deliberately. If you want to prove a token can actually be obtained, that is the layer 3 check.
Our registry is only reachable inside the VPC. Can CronAlert reach it?
Not directly; checks come from the public internet. Publish it through a Cloudflare Tunnel with an Access service token in the monitor's headers, or skip the inbound checks and rely on the in-cluster pull canary, the replication heartbeat, and the GC heartbeat, which are outbound and cover more real failures than the /v2/ check does anyway.
What about GitLab's, Gitea's, or GitHub's registries?
GitLab and Gitea embed a distribution registry, so the /v2/ expected-401 check and the pull canary apply unchanged; the GitLab and Gitea guides cover the rest of each. GHCR, Docker Hub, and ECR are services you do not run; monitor them as dependencies, with your pull canary as the check that matters.
Can the pull canary run outside Kubernetes?
Yes. On a Docker host, a cron that runs docker pull registry.example.com/canary/ping:latest && curl -fsS https://cronalert.com/api/heartbeat/… is the same check, and crane pull or skopeo inspect do it without a daemon. Run it where your deploys actually happen.
Which plan do I need?
The /v2/, Harbor health, token-grant, and manifest monitors are HTTP checks with custom headers and an expected status, on every plan at 3-minute intervals on Free and 1-minute on Pro, and SSL expiry monitoring is free. Keyword assertions and the replication, GC, and pull-canary heartbeats are Pro ($5/mo, or $4/mo billed annually).
Monitor the pull, not the port
A registry's outage is paid for later, by whoever deploys next. One expected-401 monitor, Harbor's health endpoint, a token grant, SSL expiry, and a pull canary inside the cluster turn that deferred surprise into an alert with a layer attached. Create a free account, add the /v2/ and health monitors, and upgrade to Pro for the pull canary before the next node reboots into ImagePullBackOff. Related reading: Kubernetes uptime monitoring, monitoring Jenkins for the pipelines that push, monitoring Docker and self-hosted apps, microservices monitoring for what the images become, and choosing between HTTP, TCP, and heartbeat monitors.