What does Persistent=true do in a systemd timer?

With Persistent=true, systemd records the last time the timer fired, and if the machine was down (or the timer stopped) when a scheduled run was missed, the service runs immediately on the next boot or timer start. Without it, missed runs are simply skipped — a nightly backup server that reboots at 03:00 quietly loses that night's backup. Set Persistent=true on any timer whose job must not be skipped, and use a heartbeat so a run that is missed anyway becomes an alert instead of a surprise.

How to Monitor systemd Timers and Services with Heartbeats

Q: Why didn't my systemd timer run?

The usual suspects: the timer was never enabled (systemctl enable --now name.timer — installing the unit file is not enough), the server was off or rebooting at the scheduled moment and the timer lacks Persistent=true, an OnCalendar expression doesn't mean what you think (validate with systemd-analyze calendar), or a previous run of the service is still active and Type=oneshot won't start a second instance. systemctl list-timers shows the next and last trigger times — but nobody looks at that on a schedule, which is why a heartbeat that alerts on silence is the reliable answer.

Q: Does systemd notify me when a service or timer fails?

Not by itself. A failed oneshot service just sits in a failed state in systemctl --failed and the journal; no email, no webhook, nothing. The built-in hook is OnFailure=, which starts another unit when the service fails — you point it at a small unit that curls a webhook or sends an alert. But OnFailure only fires when the service runs and fails; it cannot tell you about the more insidious case where the timer never triggered at all. Pair OnFailure (loud failures) with a success heartbeat (silence detection) to cover both.

Q: How do I add a heartbeat ping to a systemd service?

Add ExecStartPost=/usr/bin/curl -fsS --retry 3 https://cronalert.com/api/heartbeat/TOKEN to the [Service] section of the oneshot unit. For Type=oneshot, ExecStartPost commands run only after every ExecStart command has succeeded — so a failed job sends no ping, and the missing ping is what triggers the alert. This is more robust than appending && curl inside the ExecStart script, because it survives script rewrites, and cleaner than a wrapper script. Never put the ping in ExecStopPost, which runs on failure too and would mask it.

Q: Are systemd timers better than cron?

They fix real cron problems: per-job logging in the journal, dependency ordering, resource limits, RandomizedDelaySec to avoid thundering herds, Persistent=true for missed runs, and no environment-variable surprises. But they do not fix the observability problem — a timer that never fires or a service that fails is exactly as silent as a broken crontab. Whichever scheduler you use, the monitoring pattern is identical: ping a heartbeat URL on verified success, and let silence raise the alert.

systemd timers are what cron grows up into: journald logging, dependency ordering, resource limits, catch-up runs after downtime. Most modern distros schedule their own maintenance with them, and more teams are migrating their backup and sync jobs from crontab to .timer units every year. What the migration doesn't fix is the part that was always broken: when a scheduled job stops running, nothing tells you. A failed oneshot service sits quietly in systemctl --failed; a timer that was never re-enabled after a rebuild fires never; a machine that was rebooting at 03:00 skips the nightly backup without a word — unless you configured Persistent=true, and even then only sometimes.

This guide covers how systemd-scheduled jobs actually fail, the unit-file settings that prevent half of it, and the heartbeat pattern that catches all of it — including the failures the unit files can't prevent. Copy-paste unit files included.

The ways a systemd-scheduled job silently dies

The timer was never enabled. Dropping backup.timer into /etc/systemd/system/ does nothing until systemctl enable --now backup.timer. Config-management runs, server rebuilds, and restores from image are where this bites — everything looks deployed.
Missed runs with no catch-up. Without Persistent=true, a run scheduled while the machine was down is skipped, not deferred. Reboot windows and nightly jobs love to overlap.
The service failed and stayed failed. A oneshot that exits non-zero lands in systemctl --failed and the journal — and nowhere else. There is no built-in notification of any kind.
OnCalendar doesn't mean what you thought. OnCalendar=Mon *-*-* 03:00 vs OnCalendar=Mon..Fri 03:00 — always validate with systemd-analyze calendar "expression", which prints the next trigger times.
The previous run never finished. A hung job holds the service active; the next timer trigger does nothing because the unit is already running. A stuck NFS mount can stall a nightly job for weeks this way.
The job ran but the work failed. The script exited 0 while the backup uploaded zero bytes. Exit codes are claims, not proof — our backup monitoring guide is entirely about this gap.

systemctl list-timers displays last-run and next-run times for every timer — a genuinely good debugging tool that nobody opens until after the incident. Detection needs to be push, not pull.

The heartbeat pattern, systemd-native

The fix is the same silence-is-the-alarm pattern as for cron, Kubernetes CronJobs, and GitHub Actions schedules: the job pings a unique CronAlert heartbeat URL after each verified success, CronAlert knows the expected interval, and a missing ping becomes an alert on every channel your team uses. systemd gives it an unusually clean home — ExecStartPost:

# /etc/systemd/system/backup.service
[Unit]
Description=Nightly database backup
# Loud failures: fire the alert unit if this service fails
OnFailure=alert-failure@%n.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
# Runs ONLY if ExecStart succeeded — silence on failure is the point
ExecStartPost=/usr/bin/curl -fsS --retry 3 https://cronalert.com/api/heartbeat/TOKEN

# /etc/systemd/system/backup.timer
[Timer]
OnCalendar=*-*-* 03:17:00
# Run on next boot if the scheduled run was missed
Persistent=true
# Spread load if many machines share the schedule
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

Why ExecStartPost and not && curl inside the script? For Type=oneshot, ExecStartPost runs only after every ExecStart succeeded — same semantics, but it survives script rewrites and is visible in the unit file where the next admin will look. Do not use ExecStopPost for the ping: it runs on failure too, and a ping-on-failure masks exactly what you're trying to catch. And as with GitHub Actions, size the heartbeat's grace period to absorb RandomizedDelaySec plus normal job duration.

OnFailure: the loud half of the pair

Heartbeats detect silence — but when a service fails outright, you can know in seconds instead of a grace period. OnFailure= starts another unit whenever the service enters a failed state; make it a templated alert unit and every job on the box can share it:

# /etc/systemd/system/[email protected]
[Unit]
Description=Report failed unit %i

[Service]
Type=oneshot
# Any webhook works: Slack, Discord, or a CronAlert-monitored endpoint
ExecStart=/usr/local/bin/report-failure.sh %i

The two mechanisms cover different failure shapes and are worth running together: OnFailure fires the moment a run fails (but says nothing when the timer never triggers); the heartbeat catches never-ran, still-running, and machine-gone (but only after the grace period). Loud failures fast, silent failures guaranteed.

Don't forget the services the timers feed

Timers are one half of a systemd fleet; long-running services are the other, and Restart=always hides their crashes beautifully — a service crash-looping every 40 seconds looks "active (running)" at every glance. Two complements:

HTTP checks on what the box serves. If the service answers HTTP, point a monitor at its health endpoint — a crash loop shows up as flapping, and a hung process as timeouts. Our self-hosted monitoring guide covers the reverse-proxy wrinkles.
A whole-machine heartbeat. One extra timer that pings a heartbeat every 5 minutes turns "the server is up and running its schedule" into a monitored claim — cheap dead-man's-switch coverage for hosts that don't serve HTTP at all, same pattern as our IoT device monitoring guide.

Set it up in ten minutes

Create a CronAlert account — heartbeat monitors are on every paid plan, from $5/mo Pro.
Create one heartbeat monitor per timer-driven job; set the expected interval to the OnCalendar cadence and grace to RandomizedDelaySec + typical duration + margin.
Add ExecStartPost=curl … to each oneshot service; add Persistent=true to each timer while you're in the file.
Add OnFailure=alert-failure@%n.service for the fast-notification half.
systemctl daemon-reload, then verify each timer with systemd-analyze calendar and a manual systemctl start backup.service — confirm the ping arrives.
Test the failure path: make the script exit 1, confirm no ping goes out and the alert fires after the grace period.

Frequently asked questions

Why didn't my systemd timer run?

Most often: never enabled, machine down at the trigger without Persistent=true, an OnCalendar expression that means something else (check systemd-analyze calendar), or the previous run still active. A heartbeat converts all of them into the same alert.

Does systemd notify me when a service or timer fails?

No — failures sit in systemctl --failed and the journal. OnFailure= gives you a hook for loud failure alerts; only a heartbeat catches the timer that never fired.

What does Persistent=true do?

Runs a missed schedule at the next opportunity (e.g. after boot) instead of skipping it. Set it on any timer whose job must not be silently skipped.

How do I add a heartbeat ping to a systemd service?

ExecStartPost=/usr/bin/curl -fsS --retry 3 https://cronalert.com/api/heartbeat/TOKEN — for oneshot units it runs only on success. Never use ExecStopPost, which also runs on failure.

Are systemd timers better than cron?

Operationally yes — logging, ordering, catch-up runs, jitter. Observably no: they fail just as silently, and the heartbeat pattern is identical.

Give your timers a witness

Every failure mode above ends the same way: the work stops and the box keeps quiet. Two lines per unit file — ExecStartPost for the heartbeat, OnFailure for the klaxon — and a heartbeat monitor per job turn systemd's silence into alerts wherever your team lives. Create a free CronAlert account and wire up your first timer on the $5/mo Pro plan in the next ten minutes.