Why didn't Airflow's on_failure_callback fire when my DAG stopped running?

Because callbacks run when a task or DAG run finishes in a failed state — they need a run to exist. A DAG that is paused, that failed to import because of a syntax error or missing dependency, that is starved of pool slots, or whose scheduler process has died produces no runs, so there is nothing to fail and no callback to fire. Callbacks also execute inside the worker or scheduler process; if that process is OOM-killed mid-task, the callback dies with it. Silence-based monitoring is the only thing that catches a DAG that never ran.

How do I know if the Airflow scheduler is running?

Two independent ways. Airflow's webserver exposes a /health endpoint whose JSON reports the scheduler's latest heartbeat and a status of healthy or unhealthy — monitor it with an HTTP check, but configure the keyword check to alert when the body contains 'unhealthy', because a check for 'healthy' also matches 'unhealthy'. Second, run a canary DAG every five minutes whose only task pings an external heartbeat URL; if the pings stop, the scheduler, executor, or workers are not doing their job, whatever the dashboard says.

How to Monitor Apache Airflow with Heartbeats: Paused DAGs, Import Errors, and a Dead Scheduler

Q: What is a canary DAG?

A trivial DAG on a short schedule — every five minutes is typical — with one task that does nothing except call a heartbeat URL. It exercises the entire scheduling path: the scheduler parsing DAGs and creating runs, the executor queuing the task, a worker picking it up and running it, and outbound network from the worker. If any layer stalls, the heartbeat monitor alerts within minutes, and because the canary is independent of your real DAGs, a silent canary next to a silent business DAG tells you to look at the platform rather than the DAG.

Q: Does this apply to managed Airflow like MWAA, Cloud Composer, or Astronomer?

Yes, and arguably more so. Managed services handle the scheduler process for you, but paused DAGs, import errors, pool starvation, and environment updates that leave workers stuck are still your problem, and the provider's health status describes their infrastructure rather than whether your DAGs are running. The canary DAG works unchanged on all three; the /health endpoint may be behind the provider's authentication, in which case the canary is your external signal.

Airflow gives you excellent tools for knowing when a DAG run fails: on_failure_callback, task retries, email on failure, the red squares in the grid view. Our batch-job guide covers instrumenting a DAG run with start and finish heartbeats, and that pattern is right for the runs that happen. This post is about the runs that don't. A DAG that never starts produces no failed state, fires no callback, and turns no square red. It just stops appearing in the grid, and the grid is very good at not drawing attention to what isn't there.

The failures below are scheduler-level: they stop DAGs from running at all, and every one of them is invisible to run-level instrumentation. The fix is a heartbeat from a canary DAG that proves the whole scheduling path is alive, plus a correctly configured check on Airflow's own health endpoint. Both work on self-hosted Airflow and on MWAA, Cloud Composer, and Astronomer.

Why callbacks can't see these

Callbacks fire when a task instance or DAG run transitions into a terminal state. They require a run to exist. Everything in the next section prevents the run from existing, or kills the process the callback would have run in. That's the whole argument in two sentences, and it's why teams with thorough on_failure_callback coverage still discover a week later that the nightly revenue DAG hasn't run since Tuesday.

The six ways a DAG stops running silently

1. It's paused

Paused is a valid, deliberate state. Someone pauses the DAG during an incident to stop it making things worse and forgets to unpause. New DAGs deploy paused by default (depending on your dags_are_paused_at_creation setting) and someone assumes the deploy turned it on. A paused DAG sits in the list with its toggle off, generating nothing, and nobody is notified because nothing is wrong.

2. It failed to import

A syntax error in the DAG file, an import of a library that isn't in the new image, a top-level call that raises — and the DAG vanishes from the scheduler. The webserver shows a "DAG import errors" banner that everyone has learned to scroll past; the scheduler logs the traceback; and the DAG's schedule simply stops being evaluated. Callbacks can't fire for a DAG that doesn't exist. This one is especially nasty after dependency upgrades, because the DAG that breaks is often not the one you changed.

3. The scheduler died

The scheduler is a single process (or a small number of them) that parses DAGs and creates runs. If it's OOM-killed, deadlocked on the metadata database, stuck on a slow DAG parse, or simply not restarted after a host reboot, every DAG stops. The webserver keeps serving the UI from the database, so the grid looks normal — it just stops gaining new columns. Managed services restart the process for you, but "restarting" and "healthy" are not the same word, and a scheduler in a crash loop schedules nothing.

4. The workers or executor stalled

The scheduler creates the run and queues the task; then nobody picks it up. Celery workers disconnected from the broker, Kubernetes executor pods that can't be scheduled because the node pool is full, a managed environment mid-update. Tasks sit in queued indefinitely. The DAG run is "running," which is technically true and completely misleading — and because the task never enters a terminal state, no callback fires. (Airflow does eventually detect some of these as zombies and fail them, but "eventually" is measured in scheduler config, not in your SLA.)

5. Pool or concurrency starvation

A pool with eight slots, a backfill that grabbed all eight, and your critical hourly DAG's task queued behind it for six hours. Or max_active_runs reached because an earlier run is stuck. Nothing failed; the work just didn't happen when it needed to. The run-level start/finish pair from the batch-job guide catches this as "started, never finished" — if the start ping's task got a slot, which in a starved pool it may not have.

6. Catch-up ran the wrong thing

The inverse failure. Unpause a DAG that's been off for a week with catchup=True and the scheduler dutifully creates a run for every missed interval — sometimes hundreds — each processing a stale partition, sending stale emails, or overwriting current data with old. The DAG ran fine. It just shouldn't have. This is a heartbeat-visible event too: a burst of pings where there should be one is as diagnostic as silence, which is why some teams monitor critical DAGs with a max-runs-per-window check alongside the min.

The canary DAG

One DAG, one task, one job: prove the scheduling path works end to end. Every five minutes, ping an external heartbeat monitor. If the pings stop, the scheduler, the executor, the workers, or outbound network from the workers is broken — whichever it is, you have minutes of warning instead of a week.

from datetime import datetime, timedelta
import urllib.request
from airflow.decorators import dag, task

@dag(
    dag_id="canary_heartbeat",
    schedule="*/5 * * * *",
    start_date=datetime(2026, 1, 1),
    catchup=False,             # a burst of catch-up pings would hide a real gap
    max_active_runs=1,
    is_paused_upon_creation=False,
    default_args={"retries": 0},
    tags=["monitoring"],
)
def canary_heartbeat():
    @task(execution_timeout=timedelta(seconds=30))
    def ping():
        urllib.request.urlopen(
            "https://cronalert.com/api/heartbeat/CANARY_TOKEN", timeout=10
        )
    ping()

canary_heartbeat()

Configure the heartbeat monitor to expect a ping every five minutes with five to ten minutes of grace. Now walk back through the six failures. Paused (if the canary itself is paused, you'll know within ten minutes — and if only a business DAG is paused, see below). Import error in the canary file: pings stop. Scheduler dead: pings stop. Workers stalled: pings stop. Pool starvation: give the canary its own one-slot pool so it can't be starved by a backfill — then its pings prove the executor works while a business DAG's start-ping silence proves the pool is the problem. Catch-up storm: catchup=False keeps the canary honest.

The canary answers "is Airflow scheduling?" It doesn't answer "did the revenue DAG run?" For that, each critical DAG needs its own heartbeat via on_success_callback — the batch-job guide has the code — and then the two signals together are diagnostic: silent DAG, live canary means look at that DAG (paused? import error? pool?); silent DAG, silent canary means look at the platform.

The /health endpoint, without the trap

Airflow's webserver exposes /health, returning JSON that includes the metadata database status and the scheduler's latest_scheduler_heartbeat with a status of healthy or unhealthy. If the endpoint is reachable from outside (on self-hosted deployments it often is; on managed services it may sit behind the provider's auth), point an HTTP monitor at it — and configure the keyword check carefully. A check that the body contains healthy passes on "status": "unhealthy", because unhealthy contains healthy. Use CronAlert's "does not contain" match on the keyword unhealthy instead: the monitor is up while the word is absent, and alerts the moment it appears. It's a small thing that has produced a surprising number of confidently green dashboards.

Note what /health does and doesn't cover: it reports the scheduler heartbeat and the database, not whether workers are picking up tasks or whether any particular DAG is parsing. It's a good second signal and a poor only one. The canary covers the rest.

Two smaller checks worth adding

Import errors as a DAG. A second small DAG that queries the metadata database for rows in the import-errors table (the model is ImportError in Airflow's ORM; exact access varies by version) and pings a heartbeat only when the count is zero turns the banner nobody reads into an alert somebody gets. Alternatively, fail your CI on airflow dags list-import-errors before the DAG ever deploys.
SLA misses, cautiously. Airflow 2's sla and sla_miss_callback have a reputation for firing unreliably, and Airflow 3 removed the feature pending a replacement (check current docs for your version). If you're relying on SLA misses as your "DAG didn't run" detector, you're relying on the mechanism this whole post exists to back up.

Testing it

Pause the canary. Wait one interval plus grace. Confirm the alert arrives. Unpause. This is the cheapest end-to-end test of the entire chain and worth doing once so you believe it.
Break the import. On a non-production environment, push a DAG file with a syntax error and watch the DAG vanish from the list without a callback. Then watch the import-error check catch it.
Stop the scheduler on a staging deployment and confirm the canary goes silent within ten minutes while the UI keeps looking fine. Nothing makes the argument for external monitoring faster than watching a green grid stop growing. A heartbeat you've never seen fail is a hope, not a system.

Set it up in fifteen minutes

Create a CronAlert account — heartbeat monitors are on Pro at $5/mo (100 monitors); the /health HTTP monitor fits on the free plan.
Deploy the canary DAG with its own one-slot pool; heartbeat monitor at 5 minutes plus grace.
Add on_success_callback heartbeats to the handful of DAGs whose silence would cost you a day — the batch-job guide has the code and the start/finish pattern for long runs.
Monitor /health with a "does not contain unhealthy" keyword check, if it's reachable.
Pause the canary once to prove the alert arrives.

Frequently asked questions

Why didn't on_failure_callback fire?

Callbacks need a run in a failed state. Paused, un-importable, starved, or unscheduled DAGs produce no runs — and a worker killed mid-task takes the callback with it.

How do I know the scheduler is alive?

/health with a "does not contain unhealthy" check, plus a canary DAG pinging a heartbeat every five minutes. Either alone has blind spots; together they cover the scheduling path.

What is a canary DAG?

A one-task DAG on a five-minute schedule whose only job is to ping a heartbeat URL. Silence means the platform isn't scheduling, whatever the grid shows.

MWAA, Composer, Astronomer?

Same failures, same fix. The provider's health status describes their infrastructure, not whether your DAGs run. The canary works unchanged; /health may be behind their auth.

Should every DAG get a heartbeat?

No — the canary plus the few DAGs whose silence would hurt. Two hundred heartbeats is alert fatigue, not coverage.

Watch for the column that doesn't appear

Airflow is honest about failures and silent about absences, and absences are what cost you the week. A canary DAG turns "is anything scheduling?" into a heartbeat that alerts in minutes; a correctly configured health check backs it up; and per-DAG success pings tell you which layer to look at. Related reading: monitoring long-running batch jobs and DAG runs, monitoring Celery Beat (the same liveness canary, one layer down), monitoring Kubernetes CronJobs, and the heartbeat guide.