Data pipelines fail in a way that is worse than crashing: they keep looking fine. The Airbyte server is up, the webapp loads, and the connection that feeds revenue reporting has not completed a sync since Tuesday because its source credentials expired. The dbt Cloud job completed on schedule, with errors on four models, and the models downstream of them were skipped. The dashboard is green, the numbers are three days old, and the first person to notice is the CFO. None of the tools lied. Each one answered the question it was asked, and nobody asked whether the data arrived.
This guide monitors an ELT stack built on Airbyte and dbt in layers, from the tools to the data: is Airbyte serving, are syncs completing, are dbt jobs running, did they succeed, and is the output fresh. Each layer is one CronAlert monitor, and the one that matters most is the last one. The HTTP checks run on the free plan; heartbeats and content-staleness monitoring are Pro.
Layer 1: Airbyte is serving
Airbyte's server exposes a health endpoint that answers without authentication on a default self-managed install:
URL: https://airbyte.example.com/api/v1/health
Method: GET
Expected status: 200
Keyword (Pro): contains "\"available\":true" Where the path lives depends on how you deployed. On current Helm and abctl installs it is on the same hostname as the webapp. Older Docker Compose deployments route the API through airbyte-proxy, which adds HTTP basic auth, so put a Basic Authorization header in the monitor's custom headers and expect 200; a 401 means the proxy is up and the credentials changed, which is also worth knowing. Add a second monitor on the webapp root for the UI itself, since the two are different processes and the one your team notices is the UI.
Airbyte depends on a Postgres database and a Temporal server. If either is down the health endpoint stops saying available, so one keyword check covers both. If your Airbyte is only reachable inside a VPC, publish it through a Cloudflare Tunnel for the inbound checks, or skip them and rely on the heartbeats below, which are outbound and work from anywhere.
Layer 2: syncs are completing (a webhook into a heartbeat)
A healthy Airbyte with a connection that never syncs is the common failure, and the health endpoint cannot see it. Airbyte's notification system can. Under Workspace settings → Notifications, self-managed Airbyte can send webhooks for failed syncs and for successful syncs. Point the success webhook at a CronAlert heartbeat:
Airbyte → Workspace settings → Notifications
Successful syncs: webhook ON
Webhook URL: https://cronalert.com/api/heartbeat/<token>
CronAlert heartbeat monitor
Expected interval: the sync schedule (e.g. 1 hour)
Grace period: the longest normal sync, plus slack (e.g. 30 minutes) Airbyte POSTs a JSON payload with the connection, source, destination, record counts, and Slack-formatted text blocks; CronAlert's heartbeat endpoint accepts the POST, records the arrival, and ignores the body. Every successful sync in the workspace pings the heartbeat, so if the workspace goes quiet, the heartbeat goes overdue and you hear about it.
Be precise about what that proves. Notifications are per workspace, so this heartbeat means "some connection in this workspace succeeded recently", not "the revenue connection succeeded". For a workspace with one critical connection on an hourly schedule and a few daily ones, it is still useful: the hourly one dominates the pings and its absence shows up within the grace. For a per-connection guarantee, ask Airbyte directly on a schedule and ping a heartbeat per connection:
#!/bin/sh
# Ping only if the connection's latest succeeded job finished in the last 2 hours
LAST=$(curl -fsS --max-time 20 -u "$AIRBYTE_USER:$AIRBYTE_PASS" \
"https://airbyte.example.com/api/public/v1/jobs?connectionId=$CONN&status=succeeded&limit=1&orderBy=updatedAt%7CDESC" \
| jq -r '.data[0].updatedAt')
[ -n "$LAST" ] && [ "$(date -u -d "$LAST" +%s)" -gt "$(( $(date +%s) - 7200 ))" ] \
&& curl -fsS --max-time 10 --retry 3 https://cronalert.com/api/heartbeat/<revenue-token> Run it every 15 minutes from a host that is not the Airbyte server, with the heartbeat's expected interval at 15 minutes and a grace of 15. The failure webhook, meanwhile, should keep going to Slack or email; it is a different signal and does not belong on a heartbeat. If an orchestrator such as Airflow or Dagster triggers your syncs, the orchestrator's own heartbeat covers scheduling; monitoring Airflow covers the scheduler-level failures that stop syncs from being triggered at all.
Layer 3: dbt Cloud jobs are running (a webhook per job)
dbt Cloud's webhooks live under Account settings → Webhooks. A subscription can be limited to specific jobs and to specific events: job.run.started, job.run.completed, and job.run.errored. For each job whose silence would hurt, create one subscription:
dbt Cloud → Account settings → Webhooks → Create
Name: Heartbeat: nightly marts
Events: job.run.completed
Jobs: nightly_marts (one job per webhook)
Endpoint: https://cronalert.com/api/heartbeat/<marts-token>
CronAlert heartbeat monitor
Expected interval: 24 hours
Grace period: the longest normal run, plus slack (e.g. 2 hours) dbt Cloud POSTs a JSON payload including runStatus and runStatusCode, retries failed deliveries five times, and expects a response within 10 seconds; the heartbeat endpoint answers in well under a second. job.run.completed fires only after the run's artifacts are available, so it arrives a little after the run's own finish time; the grace absorbs that.
The honest caveat: job.run.completed fires for failed runs too. This heartbeat means "the job ran on schedule", which catches the schedule being paused, the environment broken, dbt Cloud's scheduler delayed, and the job deleted. It does not catch a run that completed with errors. Two ways to get the second half. The simple one: keep dbt Cloud's built-in job notifications for failures on email or Slack, and use the heartbeat for "did it run". The strict one: send the webhook to a tiny relay that reads runStatus and pings the heartbeat only on Success; the relay recipes cover a twenty-line Worker that does this kind of filtering, and monitoring the receiver covers keeping the relay itself honest. Do not subscribe job.run.started to the heartbeat: a job that starts and dies would report healthy.
Layer 4: dbt Core, pinging only on success
If you run dbt Core from cron, Airflow, or GitHub Actions, there is no webhook and no need for one. End the command with a ping that only fires on exit code 0:
0 5 * * * cd /srv/analytics && dbt build --target prod \
&& curl -fsS --max-time 10 --retry 3 https://cronalert.com/api/heartbeat/<build-token> dbt exits non-zero when any model or test fails, so this heartbeat means the whole build succeeded, which is stricter than the dbt Cloud webhook. Set a 24-hour interval and a grace covering the longest normal build. For a build that must finish by 7 a.m. for a 7:30 meeting, a 2-hour grace on a 5 a.m. schedule alerts at 7, not at 11. Batch job monitoring covers paired start/finish pings for long builds and the partial-failure trap in detail, and GitHub Actions scheduled workflows covers the case where the schedule itself drifts.
Layer 5: the data is actually fresh
Everything above monitors the pipeline's machinery. This layer monitors the pipeline's product, and it is the only layer that catches a paused connection, a source returning zero rows, a build that succeeded on stale inputs, or a dashboard pointed at the wrong schema.
Have the pipeline write one small, public, cheap artifact at the very end: a JSON file with the latest loaded_at, a status-page row, or a one-line text file in a bucket. Then monitor it with CronAlert's content staleness mode, which hashes the response and alerts when the body has not changed for a number of days:
URL: https://data.example.com/status/freshness.json
Expected status: 200
Match type: Content stale
Stale after: 1 (day) If the file stops changing because nothing upstream ran, or ran but wrote nothing new, the monitor goes red within a day of the last real update. Staleness is measured in whole days, so this is the daily-pipeline check; for an hourly pipeline, write the hour into the file and use a keyword or regex check against the current date instead, or run dbt source freshness on a schedule and ping a heartbeat only when it passes:
*/30 * * * * cd /srv/analytics && dbt source freshness --target prod \
&& curl -fsS --max-time 10 --retry 3 https://cronalert.com/api/heartbeat/<freshness-token> dbt source freshness exits non-zero when any source crosses its error_after threshold, so the heartbeat means "every source is within tolerance" and goes overdue within an hour of one falling behind. Content monitoring covers the change, stale, and regex modes; building a database health endpoint covers a slightly richer freshness route if you would rather compute it in the warehouse.
Reading the alerts
| Alert | What broke | First thing to check |
|---|---|---|
| Airbyte health down, webapp down | Server, host, or proxy | kubectl get pods or docker compose ps |
Health up but available missing | Postgres or Temporal unreachable | Those two containers; disk |
| Workspace sync heartbeat overdue, Airbyte healthy | Connections paused, source auth expired, or workers stuck | The connection's job history; worker logs |
| Per-connection heartbeat overdue, workspace heartbeat fine | That one connection | Its latest job's logs |
| dbt Cloud job heartbeat overdue | Schedule paused, environment broken, or dbt Cloud delayed | Job's run history; dbt Cloud status page |
| dbt Core build heartbeat overdue | Build failed, didn't start, or ran long | Cron log; target/run_results.json |
| Freshness content stale, everything else green | Pipeline ran but data did not move | Row counts at the source; the artifact writer |
| Freshness heartbeat overdue | A source crossed error_after | dbt source freshness output |
Alert channels in CronAlert are team-wide. If the data team wants pages and the rest of the company wants a Slack channel, put the pipeline monitors in their own team; alert routing with teams explains how.
Frequently asked questions
Can I point Airbyte's failure webhook at CronAlert too?
Not usefully. A heartbeat records arrivals, so a failure webhook would make the heartbeat look healthier, not worse. Send failures to Slack, email, or your incident tool, and keep the heartbeat for success.
Does this work with Airbyte Cloud?
The notification webhooks do; the health endpoint is Airbyte's to monitor, not yours. Use the success webhook into a heartbeat plus the freshness layer, and skip layer 1.
We use Fivetran or Stitch instead of Airbyte.
The shape is identical: wherever the tool offers a webhook on sync completion, point it at a heartbeat, and always add the freshness layer, because managed connectors fail in exactly the same quiet way.
How many monitors is this?
For a typical stack: two HTTP monitors on Airbyte, one heartbeat per critical connection or one for the workspace, one heartbeat per critical dbt job, and one freshness check. Six to eight monitors for most teams.
Which plan do I need?
The Airbyte health and webapp checks are HTTP monitors on every plan, at 3-minute intervals on Free and 1-minute on Pro. Heartbeats, keyword assertions, and content-staleness monitoring are Pro ($5/mo, or $4/mo billed annually).
Monitor the data, not just the tools
Airbyte and dbt will both tell you they are fine, and both will be telling the truth about themselves. One health check, a webhook per thing that must run, a success-only ping from dbt Core, and one staleness check on the output turn "the dashboard looks wrong" into an alert before the meeting. Create a free account, add the Airbyte checks, and upgrade to Pro for the heartbeats and the freshness monitor. Related reading: monitoring Kubernetes CronJobs if Airbyte runs on a cluster, monitoring Postgres replication lag for the warehouse side, monitoring the object store your destination writes to, monitoring automation workflows, and choosing between HTTP, TCP, and heartbeat monitors.