When application code breaks, it usually tells you. An exception is raised, a request returns 500, an error rate climbs, someone gets paged. The failure is loud because the system cannot proceed without a value it does not have.
Data pipelines have the opposite temperament. A pipeline can lose a third of its input and finish successfully, because processing fewer rows is not an error condition. The orchestrator shows green. The table has data in it. The dashboard renders, the numbers are plausible, and everyone downstream keeps making decisions. The failure surfaces weeks later when someone notices a figure that cannot be right, and by then nobody can say when it started.
This is the defining property of the discipline: the absence of data is not an exception. Every practice worth adopting follows from taking that seriously, and almost none of it looks like the monitoring you already have.
A dashboard is not a test
Most teams believe they would notice, because they have charts. They would not, for three reasons that compound.
A chart requires a human to look at it, and nobody looks at the ones that are usually fine. A chart shows aggregates, and aggregates absorb partial loss: dropping one region out of twelve moves a global total by a few percent, which is indistinguishable from a slow week. And a chart has no memory of what it should have looked like, so a metric that has been quietly wrong since March looks entirely normal in April.
What replaces the chart is a small set of assertions that run every time the data moves, and that fail the pipeline rather than colouring a pixel.
Four checks that catch most of it
In rough order of value per line of code. The first two take an afternoon and catch the majority of real incidents.
# 1. FRESHNESS the cheapest and most valuable check there is.
# Asserted on the data, not on the job. A job that ran and wrote
# nothing passes an orchestrator check and fails this one.
assert max(orders.updated_at) > now() - interval '90 minutes'
# 2. VOLUME against the same weekday, not against yesterday.
# Monday does not look like Sunday anywhere, and a band that has to
# accommodate both is too wide to catch anything.
rows_today = count(partition = today)
baseline = median(count(partition = same_weekday_last_4_weeks))
assert 0.75 * baseline < rows_today < 1.35 * baseline
# 3. DISTRIBUTION the one that catches upstream changes nothing else sees.
# Null rate, category mix, and the shape of key numerics.
assert null_rate(orders.customer_id) <= 0.001
assert set(orders.channel) <= {'web','ios','android','partner'}
assert abs(p50(orders.total) - p50_last_week) / p50_last_week < 0.20
# 4. RECONCILIATION the only check that proves correctness rather than
# plausibility. Compare against the system of record, not a copy.
src = source_db.sum(amount_cents, where day = D)
dst = warehouse.sum(amount_cents, where day = D)
assert src == dst, f"drift on {D}: {src - dst} cents"The fourth is the one teams skip because it is the most work, and it is the only one that would have caught the incident you are eventually going to have. Freshness, volume, and distribution all detect that something changed. Reconciliation detects that something is wrong, which is a different and stronger claim. For anything that touches money, entitlements, or a regulatory report, it is not optional.
Why row counts lie
The most dangerous class of bug produces a table with the right number of rows and the wrong contents, which is why count-based checks give false comfort.
A join that fans out because a supposedly unique key is not unique produces more rows, and every downstream sum is inflated by a factor nobody notices because revenue going up is not suspicious. A backfill that runs twice without idempotency doubles a partition, and if the doubling covers a full period the ratios all stay internally consistent. A timezone mismatch between extraction and partitioning shifts a slice of every day into its neighbour: totals over a month are perfect, and every daily figure is wrong.
Two assertions catch nearly all of it, and they belong on every model that has a grain:
# State the grain explicitly, then enforce it. If a model cannot say
# what one row means, that is the bug, and everything after it is noise.
assert count(*) == count(distinct order_id, line_no)
# Sum against the source, not just count. Fan-out and duplication both
# preserve plausibility in counts and destroy it in sums.
assert warehouse.sum(amount_cents) == source.sum(amount_cents)Backfills, and the word “again”
The test of a pipeline is not whether it runs. It is whether running it twice leaves the same result as running it once. If the answer is no, then every recovery from every future incident carries its own risk of making things worse, and the team will hesitate at exactly the moment speed matters.
Idempotency is not a sophisticated property. It comes from writing whole partitions rather than appending rows, and from keying on something stable in the source rather than on arrival order.
# Not idempotent: a second run duplicates the day, silently.
INSERT INTO fct_orders SELECT ... WHERE day = '2026-07-24';
# Idempotent: the partition is replaced atomically. Rerun freely.
BEGIN;
DELETE FROM fct_orders WHERE day = '2026-07-24';
INSERT INTO fct_orders SELECT ... WHERE day = '2026-07-24';
COMMIT;
# Better where the engine supports it: write to a new partition and swap,
# so readers never see a window in which the day is missing.Then rehearse it. Pick a day, rerun it, and assert the table is byte-identical. Do it as a scheduled job rather than a good intention, because idempotency decays the first time someone adds a step that appends to an audit log.
The contract problem is a people problem
Most data incidents originate outside the data team. A service renames a column, changes an enum, starts sending nulls in a field that was previously always populated. None of it was reckless: the engineer who made the change had no way of knowing that four dashboards and a finance report depended on the old shape, because nothing in their repository said so.
“Tell the data team before you change things” is not a solution. It relies on memory, scales badly, and fails silently. What works is making the dependency visible in the place where the change is made, which means a schema check that runs in the producing service’s own pipeline and fails their build, not yours.
# Lives in the producer's repo, runs in the producer's CI.
# The point is not the format. It is the location.
event: order.created
version: 3
owner: checkout-team
consumers: [warehouse.fct_orders, finance.monthly_close]
guarantees:
order_id: {type: string, required: always, unique: true}
amount_cents: {type: integer, required: always, min: 0}
channel: {type: enum, values: [web, ios, android, partner]}
customer_id: {type: string, required: always}
# Adding a field is fine. Removing one, retyping one, or adding an enum
# value is a breaking change and fails the check. The consumer list is
# there so the person making the change can see who they are about to
# page, before they merge rather than after.On-call, and what an SLA on data means
If nobody is paged when a table goes stale, the checks are documentation. That does not mean paging on everything: it means deciding, per dataset, what a consumer is actually entitled to, and being willing to say out loud that most datasets are not urgent.
- Tier one, feeding money movement, entitlements, or an external report. Reconciled daily, freshness paged, and a documented answer to “what do we do if it is wrong.” This tier should be small enough to list from memory.
- Tier two, operational dashboards people use to make decisions this week. Checked, alerted into a channel during working hours, not paged at night.
- Tier three, exploratory and everything else. Checks still run, failures still logged, nobody is woken up. Being explicit that this tier exists is what protects the first two from being drowned.
And publish the freshness. A visible timestamp on every dashboard, showing when the data underneath it was last verified rather than when the page was loaded, does more for trust than any amount of internal tooling. It also changes behaviour: people stop asking whether the numbers are current, because they can see it, and they start noticing when the answer is three days ago.
The question to ask on Monday
Not “is our data good.” Unanswerable, and it invites reassurance. Ask instead: if our most important table silently lost 20% of its rows today, what would tell us, and how long would it take?
If the honest answer is “eventually someone would query it and think the number looked low,” you do not have data quality problems yet. You have data quality problems and no way to know about them, which is a different and considerably worse position. The four checks above take a week to implement and turn that answer into a number of minutes.