How to Monitor Scheduled Database Backups with Heartbeats

Q: How do I know if my database backups are actually running?

Put a heartbeat monitor on the backup job: the script makes an HTTP request to a unique CronAlert heartbeat URL as its final step, only after the dump has been created and verified. CronAlert expects a ping on your backup schedule (say, daily with a few hours of grace) and alerts you the moment one fails to arrive. This catches every failure mode at once — a cron daemon that never fired, a dump that exited non-zero, a verification step that failed — because they all produce the same symptom: silence.

Q: How do I monitor pg_dump or mysqldump cron jobs?

Run the dump inside a script with set -euo pipefail, verify the artifact (file exists, size within expected range, gzip -t passes, pg_restore --list succeeds for custom-format dumps), then chain a curl to a CronAlert heartbeat URL as the last command. Because each step only runs if the previous one succeeded, the ping is proof of a verified backup, not just proof that cron fired. If any step fails, the ping never happens and CronAlert alerts you.

Q: Why does my backup script report success when the dump failed?

The most common cause is piping pg_dump through gzip without pipefail: in a pipeline, the shell's exit code is the last command's, so if pg_dump dies mid-stream but gzip exits cleanly, the pipeline reports success and you archive a truncated dump. Other culprits: scripts that use ; instead of && between steps, scripts that don't check exit codes at all, and dumps that succeed against the wrong or empty database after a migration. Fix it with set -euo pipefail and an artifact-verification step before any success signal.

Q: How often should I test database restores?

Monthly is a reasonable floor for most teams; weekly if the database is business-critical. Automate it: a scheduled job restores the latest dump into a scratch database, runs a smoke query (row counts on key tables, a recent-record check), and pings a separate CronAlert heartbeat monitor configured with a monthly interval. An untested backup is a hope, not a backup — the restore test is the only step that proves the whole chain, and giving it its own heartbeat means you get alerted when restore testing quietly stops happening.

Q: Do managed database backups like RDS or Cloud SQL need monitoring?

Yes. Managed snapshots are good but not sufficient: they can silently stop when storage quotas fill or snapshot limits are hit, they usually live in the same region and account as the database, and they don't protect you from account-level problems. Most teams still run logical dumps (pg_dump to S3, cross-region copies) for portability and disaster recovery — and those are ordinary cron jobs with all the ordinary failure modes, so they deserve heartbeat monitoring just like a self-hosted backup.

Every engineer knows a version of this story. A database dies, the team calmly reaches for the backups, and discovers the most recent one is from eleven weeks ago — the night someone rotated a database password, or the disk filled, or the server was rebuilt and the crontab never made the trip. Backups are the canonical silently-failing cron job: when one breaks, nothing user-facing changes, no error surfaces anywhere anyone looks, and the failure stays invisible until the exact moment you need it not to be.

We've covered the general pattern for catching silent job failures in our guide to cron job heartbeat monitoring. This post applies it to the job where the stakes are highest — and where "the job ran" and "you have a restorable backup" are very different claims. The goal is a backup pipeline where the success signal is only sent after the backup has been verified, so that silence means trouble and a green monitor means something real.

The ways backups fail silently

A backup pipeline is a chain of steps, and every link can fail without making a sound:

The job never fires. The server was rebuilt and the crontab wasn't restored, the cron daemon died, or a deploy replaced the container and the schedule with it. Nothing runs, so nothing errors.
The dump exits non-zero, but the script shrugs. An expired password, a full disk, a lock timeout — pg_dump or mysqldump fails, and a script without error handling carries on to report success.
The dump is truncated or empty. The disk filled mid-dump, or the connection dropped. A file exists, it just doesn't contain your database.
Compression corrupts the archive. The bytes are there; gunzip will never read them.
The offsite upload fails. The local dump is fine, but expired credentials or a renamed bucket mean the copy that survives a dead server never happened.
Retention cleanup deletes too much. An off-by-one in the pruning script and your "30 days of backups" is one day of backups.
The backup targets the wrong database. After a migration or a config change, the job faithfully dumps an empty schema every night.

Notice that most of these produce a job that appears to run. Checking "did cron fire?" catches only the first one. That's why the monitoring signal has to be tied to verified output, not to execution.

The core pattern: ping only on verified success

Heartbeat monitoring inverts the usual direction of uptime checks: instead of CronAlert reaching into your infrastructure, your job reaches out. Each backup job gets a unique heartbeat URL, and pings it when — and only when — it has succeeded. CronAlert expects a ping on your schedule and alerts you when one doesn't arrive.

The entire pattern lives or dies on where you put the ping. Here's the version that quietly lies to you:

# BAD: the ; means the ping fires whether or not the dump succeeded
0 2 * * * pg_dump mydb | gzip > /backups/mydb.sql.gz ; curl -s https://cronalert.com/api/heartbeat/TOKEN

Two failures hide in that one line. The ; runs the curl unconditionally, so a failed dump still reports success. And even with &&, the pipeline's exit code is gzip's, not pg_dump's — if the dump dies mid-stream while gzip exits cleanly, the shell calls it a success and you archive a truncated file. The fix is a real script with set -euo pipefail, verification before any success signal, and the ping as the final chained step:

#!/usr/bin/env bash
set -euo pipefail   # -e: stop on error; pipefail: pg_dump failures aren't masked by gzip

OUT=/backups/mydb-$(date +%F).sql.gz

pg_dump mydb | gzip > "$OUT"

# Verify before declaring victory
gzip -t "$OUT"                                      # archive is readable
SIZE=$(stat -c%s "$OUT")
[ "$SIZE" -gt 1000000 ] || { echo "dump suspiciously small"; exit 1; }

aws s3 cp "$OUT" s3://my-backups/mydb/              # offsite copy, fails loudly

# Only reachable if every step above succeeded
curl -fsS https://cronalert.com/api/heartbeat/TOKEN > /dev/null

Because set -e aborts the script on the first failing command, the curl at the bottom is only reachable when the dump ran, the archive verified, and the offsite copy landed. The ping stops being "the job ran" and becomes "a verified backup exists offsite" — which is the claim you actually care about. This is the same discipline we recommend for any batch job: tie the success signal to the outcome, not the attempt.

Verification tiers: from exit codes to restore tests

How much verification belongs between the dump and the ping? Think of it as tiers, each catching failures the previous one can't:

Tier 1 — exit codes. set -euo pipefail and && chains. Free, and catches outright command failures. This is the minimum.
Tier 2 — artifact sanity. The file exists, gzip -t passes, and the size is within an expected range. A good trick: fail if today's dump is smaller than 80% of yesterday's — that one comparison catches truncated dumps, empty schemas after a botched migration, and a surprising number of "how did that happen" cases. For Postgres custom-format dumps, pg_restore --list proves the archive's table of contents is readable without restoring anything.
Tier 3 — offsite confirmation. Check the upload's exit code, or verify the object's size after upload. The classic 3-2-1 rule (three copies, two media, one offsite) only protects you if each copy actually happens — a local dump on the same disk as the database is not a backup plan.
Tier 4 — restore tests. On a schedule, restore the latest dump into a scratch database, run a smoke query (row counts on key tables, does the newest record look recent?), and ping a separate heartbeat monitor on success. This is the only tier that proves the whole chain end to end. An untested backup is a hope, not a backup.

Tiers 1–3 belong in every nightly backup script; they add seconds. Tier 4 typically runs weekly or monthly, and giving it its own heartbeat means you find out when restore testing itself quietly stops — which, being a cron job, it eventually will.

Setting up the heartbeats in CronAlert

The setup mirrors the tiers. In CronAlert, create one heartbeat monitor per backup job per database — backup-postgres-prod, backup-mysql-analytics — plus one for the restore test. Each monitor gets a unique URL of the form https://cronalert.com/api/heartbeat/<token>, an expected interval, and a grace period:

Nightly dump: expected every 24 hours, with a grace period of a few hours. Backup duration varies with data size, so a 2 AM job with a 4-hour grace pages you at 6 AM if the ping hasn't arrived — after any reasonable run would have finished, but hours before anyone would have noticed otherwise.
Restore test: expected weekly or monthly, with a day of grace.

When a ping goes missing, CronAlert alerts through whatever channels you've wired up: email, Slack, Discord, Microsoft Teams, Telegram, PagerDuty, Opsgenie, Splunk On-Call, webhooks, or PWA push. Heartbeat monitors are available on every paid CronAlert plan — Pro at $5/mo includes 100 monitors, which comfortably covers a whole fleet of databases, restore tests, and the rest of your background jobs besides.

One subtle benefit of the grace period: it doubles as a duration alarm. A backup that used to take ten minutes and now takes three hours — runaway table growth, a missing index on the dump path, a saturated disk — will start brushing up against the grace window before it fails outright. If you want an explicit signal earlier than that, expose an internal backup-status endpoint (last run time, duration, artifact size) and put a keyword check on it; the same idea we use for response-time thresholds applies to jobs: alert on the trend, not just the failure.

Managed databases still need this

"We're on RDS, snapshots are handled" is true right up until it isn't. Automated snapshots can quietly stop when storage quotas fill or snapshot limits are hit; they live in the same region and account as the database, which is exactly the blast radius you're trying to escape; and they don't help if the account itself is the problem. That's why most teams also run logical dumps — pg_dump to S3, cross-region copies — for portability and true disaster recovery. Those dumps are ordinary cron jobs on some EC2 box or container, with every failure mode this post opened with, and they deserve the same heartbeat treatment as a self-hosted setup. Monitor the copies you control; trust but verify the ones you don't.

Frequently asked questions

How do I know if my database backups are actually running?

Put a heartbeat on the job: the script pings a unique CronAlert URL as its final step, only after the dump is verified. CronAlert expects the ping on schedule and alerts on silence — which catches a dead cron daemon, a failed dump, and a failed verification all the same way.

How do I monitor pg_dump or mysqldump cron jobs?

Wrap the dump in a script with set -euo pipefail, verify the artifact (size range, gzip -t, pg_restore --list), then chain curl -fsS to your heartbeat URL as the last command. The ping becomes proof of a verified backup, not just proof that cron fired.

Why does my backup script report success when the dump failed?

Usually a pipeline without pipefail: pg_dump | gzip reports gzip's exit code, so a dump that dies mid-stream still looks successful and you archive a truncated file. Scripts using ; instead of &&, or ignoring exit codes entirely, fail the same way.

How often should I test database restores?

Monthly at minimum; weekly for business-critical data. Automate it — restore into a scratch database, run a smoke query, ping a separate heartbeat — so you're alerted when restore testing itself stops happening.

Do managed database backups like RDS or Cloud SQL need monitoring?

Yes. Snapshots can silently stop on quota issues, and they share a region and account with the database they protect. The logical dumps and cross-region copies you run alongside them are normal cron jobs — monitor them with heartbeats like any other.

Make silence impossible to miss

The cruelest property of a broken backup job is that it fails in the one way nothing notices. A heartbeat monitor turns that around: silence itself becomes the alarm, and a verified-success ping means every link in the chain — dump, integrity check, offsite copy — actually held. Set it up once and the eleven-weeks-of-missing-backups story can't happen to you. Create a CronAlert account, add a heartbeat monitor for each database on the Pro plan ($5/mo covers 100 monitors), and chain the ping into your backup script tonight — it's one line at the end of a script.