A self-hosted inference server is a GPU, a process, and a model, and each one fails without taking the others with it. The process is up and listening, but the model it was supposed to load failed on a CUDA error and it is serving an empty list. The model is loaded and the GPU is out of memory, so every request past the first returns a 500 or hangs. An update changed the bind address back to localhost, and the box is fine, but nothing outside it can connect. The Cloudflare Tunnel in front of it dropped and the server never knew. Or the whole thing is healthy and generation has slowed from 60 tokens a second to 4 because the card is thermal throttling, and your application's timeouts are all firing.
This guide puts three layers of monitoring on a server running Ollama, vLLM, llama.cpp's llama-server, or Hugging Face's Text Generation Inference: a liveness check on the endpoint each server exposes, a readiness check that the model you depend on is still there, and a heartbeat from a scheduled smoke test that actually runs a prompt, because that is the one thing an HTTP monitor cannot do. It also covers getting the server reachable in the first place without exposing an unauthenticated model API to the internet. If you are monitoring a hosted provider's API rather than your own server, monitoring AI and LLM API dependencies is the companion guide.
First: make it reachable, safely
Most inference servers assume a trusted network. Ollama binds to 127.0.0.1:11434 by default and has no authentication; setting OLLAMA_HOST=0.0.0.0 to make it reachable also makes it reachable by anyone who finds the port. vLLM listens on all interfaces on 8000 and authenticates only if you pass --api-key. llama-server is similar, with an optional --api-key. Before pointing an external monitor at any of them, put something in front:
- A reverse proxy with a hostname and auth. Nginx or Caddy terminating TLS on
llm.example.com, forwarding to the server, and requiring an API key or basic auth on the inference routes while leaving the health route open. Monitoring the proxy separately tells you which layer broke. - A Cloudflare Tunnel. No open ports; the tunnel daemon connects out. Protect the hostname with Access and give the monitor a service token in its headers. The tunnel guide has the exact headers.
- Tailscale. Keep the server on the tailnet for your applications, and either use Tailscale Funnel to publish the health route or rely on the heartbeat layer below, which needs no inbound access at all. Monitoring over Tailscale covers the trade-offs.
One constraint shapes the choice: CronAlert's HTTP checks run on Cloudflare and cannot fetch a bare IP address. A monitor on http://203.0.113.5:11434/ fails with a synthetic 403 on every check. If the box has no hostname and you do not want one, use a TCP port monitor (Pro) on the IP and port, which proves the port accepts connections, and lean on the heartbeat for everything else.
Layer 1: liveness, the endpoint each server exposes
Every server has a cheap URL that answers while the process is alive and does not touch the model. Monitor it with expected status 200, GET, and the default 30-second timeout:
| Server | Liveness URL | Healthy | Not healthy |
|---|---|---|---|
| Ollama | / | 200, body Ollama is running | Connection refused (process down, or bound to localhost) |
| vLLM | /health | 200 | 503 when the engine has died; refused while the container starts |
llama.cpp llama-server | /health | 200, {"status":"ok"} | 503 while loading the model, 500 if the load failed |
| Text Generation Inference | /health | 200 | Non-200 while unavailable |
vLLM's /health is served without the API key even when --api-key is set, and llama-server's is too, so these monitors need no credentials if the proxy leaves the health route open. llama-server's 503 during model load is the right answer for a monitor: a server that has been "loading" for ten minutes after a restart is a server with a broken model file or not enough VRAM, and you want to know.
On Pro, add a keyword so the check proves the right thing answered. contains "Ollama is running" on Ollama's root, or contains "ok" on llama-server's health, rules out a reverse-proxy default page that also returns 200.
Layer 2: readiness, the model is still there
Liveness says the process is up. It says nothing about whether the model your application asks for by name exists on this server. Models get removed to free disk, a redeploy pulls the wrong tag, a vLLM restart with a different --model flag serves something else, and your application gets 404s with a healthy server behind them. Each server has a list endpoint; monitor it with a Pro keyword that the model's name appears:
# Ollama: models pulled to disk (not "loaded"; see below)
URL: https://llm.example.com/api/tags
Keyword: contains "llama3.1:70b"
# vLLM, llama-server, TGI: OpenAI-compatible model list
URL: https://llm.example.com/v1/models
Headers: { "Authorization": "Bearer <api-key>" }
Keyword: contains "\"id\":\"meta-llama/Llama-3.1-70B-Instruct\""Two things not to monitor here. Ollama's /api/ps lists models currently loaded in memory, and Ollama unloads an idle model after a few minutes by default, so an empty /api/ps at 3 a.m. is normal and a monitor on it would alert every quiet night. And /v1/models on vLLM may answer before a large model has finished loading on startup, so treat this as "configured to serve the right model," not "ready to answer." Readiness in the strict sense is the next layer.
If your application depends on two models on the same server, that is two monitors on the same URL with different keywords; a monitor carries one assertion.
Layer 3: a heartbeat from a real prompt
The failure that matters most is "the server is up, the model is listed, and inference is broken or crawling." GPU out of memory after a memory leak, a driver that wedged, a card throttling at 90°C, a KV cache full of stuck requests. No GET can see it. Only a completion can, and an uptime monitor cannot send one, because HTTP monitors send headers and no request body.
So run the completion yourself, on a schedule, and report success to a heartbeat monitor. The script runs on the GPU box or any machine that can reach it, sends the smallest useful prompt with a strict timeout, checks that the answer is well-formed, and pings CronAlert only if all of that succeeded:
#!/usr/bin/env bash
# llm-smoke.sh: run every 10 minutes from cron or a systemd timer
set -euo pipefail
PING="https://cronalert.com/api/heartbeat/<token>"
# Ollama
out=$(curl -fsS --max-time 20 http://127.0.0.1:11434/api/generate \
-d '{"model":"llama3.1:8b","prompt":"Reply with the single word OK.","stream":false,"options":{"num_predict":5}}')
echo "$out" | grep -q '"response":"[^"]*OK'
# vLLM / llama-server / TGI (OpenAI-compatible): swap in instead
# out=$(curl -fsS --max-time 20 http://127.0.0.1:8000/v1/chat/completions \
# -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
# -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","max_tokens":5,"messages":[{"role":"user","content":"Reply with the single word OK."}]}')
# echo "$out" | grep -q '"content":"[^"]*OK'
curl -fsS --retry 3 "$PING" > /dev/null*/10 * * * * /usr/local/bin/llm-smoke.sh >> /var/log/llm-smoke.log 2>&1Create the heartbeat with a 10-minute expected interval and a 5-minute grace. set -e and the grep make every line a gate: if the request fails, times out at 20 seconds, or returns something that is not an answer, the script exits before the ping and the heartbeat goes overdue within about 15 minutes. The --max-time 20 is your latency alarm. A healthy 8B model answers a five-token prompt in well under a second; if it takes 20, something is wrong with the GPU even though every HTTP monitor is green. Tune that number to your hardware, and keep the smoke model small so the test itself does not evict the model your users are on.
The same heartbeat covers the host. If the box reboots and the server does not come back, if the disk fills and the script cannot write its log, or if the machine loses its network, the ping stops. Monitoring a VPS end to end extends this with a host-health script; the Proxmox guide covers the case where the GPU is passed through to a VM, and the grace-period guide covers tuning the interval and grace.
Timeouts, model swaps, and restarts
Leave the HTTP monitors' timeout at the default 30 seconds; liveness and list endpoints answer in milliseconds, and a slow one is a real signal. Resist raising it to cover model-loading time. llama-server's 503 and vLLM's refused connections during a load are correct "down" answers, and a 70B model that takes eight minutes to load after an unplanned restart is eight minutes your application was down. For planned model swaps and driver updates, put a maintenance window (Pro) on the monitors and the heartbeat instead of widening the timeout permanently. Setting timeouts has the general reasoning.
If you run the server in Docker with restart: unless-stopped, a crash loop looks like a monitor that flaps every few minutes. CronAlert requires consecutive failures before alerting on 1-minute intervals, which filters a single restart; a loop will still alert, as it should. Monitoring Docker and self-hosted apps covers the container side.
Reading the alerts
| Alert | What broke | First thing to check |
|---|---|---|
| Liveness down, connection refused | Process down, bound to localhost, or the proxy/tunnel in front | ss -ltnp for the port; OLLAMA_HOST; tunnel or proxy status |
| Liveness 503 for more than a few minutes | Model failed to load or not enough VRAM | Server log; nvidia-smi for free memory |
| Liveness green, model keyword failed | Model removed, wrong tag pulled, server started with a different model | ollama list, or the --model flag in the unit file |
| Everything green, heartbeat overdue | Inference broken or slow: OOM, driver wedge, throttling, stuck queue | nvidia-smi (memory, temperature, utilization), then the smoke log for the timing |
| Heartbeat overdue and liveness down | Host down or offline | Power, console, provider status |
Alert channels are team-wide in CronAlert, so if the ML team wants pages and the rest of the company does not, give the inference monitors their own team; alert routing with teams explains how.
Frequently asked questions
Does a 1-minute check on the health endpoint load the model or cost GPU time?
No. Ollama's root, vLLM's /health, and llama-server's /health answer from the HTTP layer without touching the model. llama-server documents that /health does not reset its idle timer or wake a sleeping model. The smoke-test heartbeat does run inference, which is why it runs every 10 minutes with a five-token limit rather than every minute.
Can CronAlert tell me tokens per second?
Not as a metric; it is an uptime monitor, not a GPU profiler. The smoke script's --max-time gives you a pass/fail latency bound, and you can log the elapsed time in the script for a trend. For throughput dashboards, scrape vLLM's or llama-server's Prometheus /metrics, and if you run Prometheus for that, give it a dead man's switch too.
Should the heartbeat script use the production model or a small one?
A small one, unless the production model is the only one loaded and you want the test to catch its eviction. Loading a second model to run the test can evict the first on a memory-constrained card, which turns the monitor into the outage. Ollama's keep_alive and vLLM's single-model design behave differently here; test it once and watch nvidia-smi.
What about Open WebUI, LiteLLM, or another gateway in front?
Monitor the gateway's own health route as a separate HTTP monitor, and keep the server-level checks too. When the gateway is down and the server is up, the alerts tell you which one to restart.
Which plan?
The liveness monitors, with headers and expected status, are on every plan at 3-minute intervals, 1-minute on Pro. Model-name keyword checks, the heartbeat smoke test, TCP checks on a bare IP, and maintenance windows are Pro ($5/mo, or $4/mo annually).
Three layers, because the GPU can't raise its hand
A port check sees the process. A keyword check sees the model list. Only a prompt sees the GPU, and only a heartbeat can carry that answer out. Together they are two HTTP monitors and a ten-line script, and they turn "the model's acting weird" from a Slack thread into an alert with a cause. Create a free account, add the liveness monitor, and upgrade to Pro for the model check and the smoke-test heartbeat before the next silent OOM. Related reading: monitoring hosted LLM APIs, monitoring MCP servers if agents sit on top of the model, API endpoint monitoring, designing health check endpoints for the application that calls the model, and choosing between HTTP, TCP, and heartbeat monitors.