A Proxmox node is the machine under all the other machines. When it goes down, every VM and container it hosts goes down with it, and so does the thing that would have told you: the Postfix relay on the node, the Gotify instance in a container on the node, the Uptime Kuma you run on the node. Proxmox's own notification system is good at reporting failures the node can see. It has no way to report that the node itself is gone, and it is quiet in exactly the way you don't want when a backup job stops being scheduled.
This guide sets up three layers of monitoring for a Proxmox VE host that don't rely on the host to announce its own death: an external check from the internet, checks on what the guests actually serve, and a heartbeat from inside the node that fails on the degraded states nothing outside can see, including a backup that hasn't succeeded since yesterday. It works for a single homelab node and for a cluster, and most of it is one shell script.
Layer 1: is the node reachable at all?
If the host has a public address (a dedicated server at Hetzner or OVH, a colo box), the first monitor is a TCP port check on SSH, port 22. It attempts a connection on every check and alerts when the connection is refused or times out, which is what a powered-off node, a kernel panic, a dead NIC, or a provider network problem all look like from outside. It works against a bare IP, which matters because plenty of Proxmox hosts have no DNS name. There's no ICMP ping option, and port 22 is a better check anyway: it proves the machine is up and a service on it is answering.
Don't open port 8006 to the internet just to monitor the web interface. If it's already reachable from outside through an allowlist, a TCP check on 8006 adds "pveproxy is listening" to the picture. If you want an actual HTTPS check on the interface, be aware that Proxmox ships with a certificate signed by its own cluster CA, so an external HTTPS monitor fails the handshake and reports a certificate error rather than a 200. The fix is Proxmox's built-in ACME support: under Datacenter → ACME register a Let's Encrypt account, then on the node under System → Certificates order a certificate for its public hostname (the HTTP challenge needs port 80 reachable; the DNS challenge plugins don't). Put the interface behind your reverse proxy on a normal hostname and port and monitor https://pve.example.com/ expecting the login page's 200. Proxmox Backup Server has the same certificate story on port 8007.
If the node is at home behind NAT, or only reachable over a VPN, skip this layer. Layer 3 covers it without opening anything.
Layer 2: is what the guests serve actually up?
Nobody cares that VM 104 is running; they care that the Nextcloud on it loads. Monitor the services, not the VMs. For anything with a web interface, that's an HTTP monitor on its public hostname through your reverse proxy, expecting a 200 and, on Pro, a keyword that only appears when the app rendered. The self-hosted stack guide and the reverse proxy guide cover the per-service pattern in detail, including what a 502 from the proxy means (the guest is down or unready) versus a timeout (the proxy itself, or the host). For VMs with their own public IPs, a TCP check on 22 per VM does what it does for the host. These checks are on the free plan when they're HTTP.
The value of having both layers is in how they disagree. Host TCP check down and every service down: the node. Host fine and one service down: that guest. Host fine and every service down: the reverse proxy or the network between them. You know where to look before you've opened a terminal.
Layer 3: a heartbeat from inside the node
The failures that make a Proxmox host miserable rarely take it offline. A VM that didn't come back after a reboot because it wasn't set to autostart. A guest that's "running" but frozen, holding its memory and doing nothing. A ZFS pool that's been degraded for three weeks. The NFS share that backups go to, silently inactive. Local storage at 97%. A two-node cluster that lost quorum and can't start anything. None of these show from outside, and all of them are one command away from inside.
A heartbeat monitor inverts the check. CronAlert gives you a URL; a script on the node runs every minute, checks everything above, and pings the URL only when every check passes. If the node dies, the pings stop. If a check fails, the pings stop. Either way CronAlert opens an incident after a couple of minutes of silence and alerts you, and the reason is in the node's journal. Nothing is exposed and nothing is installed; Proxmox is Debian with cron, curl, and every pve tool already present.
#!/usr/bin/env bash
# /usr/local/bin/pve-health.sh — run from cron every minute as root
set -uo pipefail
HEARTBEAT="https://cronalert.com/api/heartbeat/<token>"
MUST_RUN_VMS="100 101 104" # VMIDs that should always be running
MUST_RUN_CTS="200 201" # CTIDs that should always be running
AGENT_VMS="100 104" # VMs with qemu-guest-agent installed and enabled
BACKUP_MARKER=/var/lib/vz/last-backup-ok
BACKUP_MAX_AGE_MIN=1560 # 26 hours for a nightly job
fail=0
# Core services
for svc in pve-cluster pvedaemon pveproxy pvestatd; do
systemctl is-active --quiet "$svc" || { echo "$svc not active"; fail=1; }
done
# Guests that must be running
for id in $MUST_RUN_VMS; do
qm status "$id" 2>/dev/null | grep -q running || { echo "VM $id not running"; fail=1; }
done
for id in $MUST_RUN_CTS; do
pct status "$id" 2>/dev/null | grep -q running || { echo "CT $id not running"; fail=1; }
done
# Guest agent answers: catches a VM that is "running" but frozen
for id in $AGENT_VMS; do
qm agent "$id" ping >/dev/null 2>&1 || { echo "VM $id agent not responding"; fail=1; }
done
# ZFS: "all pools are healthy" (or no pools at all)
zpool status -x 2>/dev/null | grep -qE 'healthy|no pools' || { echo "zfs: $(zpool status -x | head -1)"; fail=1; }
# Storage: anything Proxmox considers inactive (dropped NFS/CIFS share, unmounted disk)
inactive=$(pvesm status 2>/dev/null | awk 'NR>1 && $3=="inactive" {print $1}')
[ -n "$inactive" ] && { echo "storage inactive: $inactive"; fail=1; }
# Root filesystem below 90%
use=$(df --output=pcent / | tail -1 | tr -dc '0-9')
[ "$use" -ge 90 ] && { echo "root disk ${use}%"; fail=1; }
# Cluster quorum, only if this node is in a cluster
if [ -f /etc/pve/corosync.conf ]; then
pvecm status 2>/dev/null | grep -qE 'Quorate:\s+Yes' || { echo "cluster not quorate"; fail=1; }
fi
# Backups: marker written by the vzdump hook below must be fresh
[ -n "$(find "$BACKUP_MARKER" -mmin -"$BACKUP_MAX_AGE_MIN" 2>/dev/null)" ] || { echo "no successful backup in ${BACKUP_MAX_AGE_MIN} min"; fail=1; }
# Ping only when everything passed; silence is the alert
[ "$fail" -eq 0 ] && curl -fsS -m 10 -X POST "$HEARTBEAT" >/dev/null
exit 0Install it with a cron entry that sends the script's output to the journal, so a silent heartbeat comes with a reason:
# /etc/cron.d/pve-health
* * * * * root /usr/local/bin/pve-health.sh 2>&1 | logger -t pve-healthWhen CronAlert alerts, journalctl -t pve-health -n 20 on the node tells you which line failed. A few notes on the checks. The guest agent ping needs qemu-guest-agent installed in the VM and the agent enabled in the VM's Options; it's the only check here that distinguishes a frozen guest from a healthy one, so it's worth the two minutes per VM. Listing must-run guests explicitly, rather than "everything that exists," means a VM you deliberately stopped doesn't page you. And the quorum check is skipped on a single node; on a cluster, run this script on every node with its own heartbeat, so you learn which node fell out. Heartbeat monitors are on the Pro plan.
Backups: freshness, not just failure
Proxmox will notify you when a vzdump job fails, if you've configured a notification target that works (the default is local mail delivery, which on most nodes goes nowhere). What it won't notify you about is a job that didn't run: the schedule was disabled while you debugged something and never re-enabled, the node was mid-reboot at 02:00, the target storage was inactive so the job aborted before it started. Those are the failures you discover the day you need a restore.
The health script above already checks a marker file's age. This hook script writes the marker. vzdump calls it at each phase of a backup job, passing the phase name as the first argument; it runs for every job when set as the global script option in /etc/vzdump.conf, or per job with --script.
#!/usr/bin/env bash
# /usr/local/bin/vzdump-hook.sh — set as "script: /usr/local/bin/vzdump-hook.sh" in /etc/vzdump.conf
phase="$1"
flag=/run/vzdump-had-failure
case "$phase" in
job-init|job-start) rm -f "$flag" ;;
backup-abort) touch "$flag"; logger -t vzdump-hook "backup of guest $3 aborted" ;;
job-abort) touch "$flag"; logger -t vzdump-hook "backup job aborted" ;;
job-end) [ -f "$flag" ] || touch /var/lib/vz/last-backup-ok ;;
esac
exit 0Make it executable and it's live for the next run. The marker moves only when a job finishes with no guest aborted, so a partial job (one VM failed, four succeeded) counts as a failure, which is what you want. Set BACKUP_MAX_AGE_MIN in the health script to your schedule plus slack: 26 hours for nightly, about 8 days for weekly. If backups go to Proxmox Backup Server, add a TCP check on its port 8007 and run a heartbeat script on the PBS host too, so a PBS that's up but has stopped verifying or pruning gets its own alert. The backup monitoring guide has the reasoning behind "ping only on verified success," and it applies here unchanged.
Reading which layer fired
| Host TCP (22) | Service HTTP checks | Node heartbeat | Most likely |
|---|---|---|---|
| Down | All down | Silent | Node offline: power, kernel panic, provider network. Start at the console or the provider's status page. |
| Up | All down | Fine | Reverse proxy or the VM it runs on, or the bridge between guests and the internet. |
| Up | One down | Fine | That guest's application. The VM is running; the service isn't. |
| Up | One down | Silent | Check the journal: a must-run VM stopped, or its agent stopped answering (frozen guest). |
| Up | All fine | Silent | Degraded but serving: ZFS, inactive storage, disk space, quorum, or a stale backup. The journal line says which. |
| Up | All fine | Silent, every morning | Backups. The marker is aging out; the job isn't succeeding. |
Reboots and false alarms
Kernel updates on Proxmox mean a reboot, and a reboot means the TCP check fails and the heartbeat goes quiet for a few minutes. Put a maintenance window over your patch slot so it doesn't page you. For a home connection that drops the odd packet, set the TCP monitor's failure threshold to 2, which with 1-minute checks still alerts inside about two minutes. Then test the whole thing once: stop a must-run VM for three minutes and confirm the heartbeat alert arrives with the right journal line, then start it and confirm the recovery.
Frequently asked questions
Why does an HTTPS check on port 8006 fail?
The default certificate is signed by Proxmox's own CA, which no external monitor trusts. Use a TCP check on the port, or issue a Let's Encrypt certificate via Datacenter → ACME and monitor through a reverse proxy.
Can CronAlert ping the host?
Not with ICMP. A TCP check on port 22 proves more anyway. For a host with no public address, use the heartbeat.
Do I need to expose anything to the internet?
No. The heartbeat script only makes outbound requests. The external TCP check is a bonus for hosts that already have a public address.
How do I know when backups stopped running?
The vzdump hook writes a marker on clean job completion; the heartbeat script fails when the marker is older than your schedule allows. Failed, skipped, and never-scheduled all look the same: silence, then an alert.
Which plan?
HTTP checks on your guests' services are free. The TCP check on the host and the heartbeat are Pro at $5/mo.
Does this work for a cluster?
Yes. Run the script on each node with its own heartbeat; the quorum check activates automatically when a corosync config is present.
The host can't report its own death
Everything Proxmox can tell you about itself arrives via Proxmox. The two things it can't tell you, that it's gone and that a job never ran, are exactly what an outside check and a heartbeat exist for. One TCP monitor, one script, one hook, and a maintenance window over patch night. Create an account, upgrade to Pro, and add the heartbeat before the next reboot you didn't schedule. Related reading: monitoring a VPS end to end for the same three-layer pattern on a plain server, monitoring Docker and self-hosted apps for what runs in the guests, TCP port monitoring, and monitoring systemd timers for the other scheduled jobs on the node.