DebuggingGENERALSIMULATEDTOOL-SPECIFIC

The Pipeline Succeeded. The Data Is Wrong.

The domain's thesis, turned into a diagnosis. A green DAG proves the code ran; eight faults, six checks and the distinct fingerprint each one leaves are what prove anything else.

What actually happens How to build it Can I trust it?▶ Run the lab

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

Every task in last night's run exited zero, inside its timeout, with no retries. What does that prove about the rows in the serving table?

Who needs this

The on-call engineer deciding whether there is an incident, and the consumer who will act on the number either way. One of them is looking at a status page and the other is looking at a figure, and nothing currently connects the two.

What one row is

Two different units are being confused, and separating them is the lesson. A task status is one row per execution — it says something about a process. Correctness is a claim about every business record in a period. No amount of the first ever aggregates into the second, because a process that handled zero records exits zero (Grain: What Does One Row Represent?).

The obvious build

Make the orchestrator the health system. Alert on task failure and on runs that exceed their expected duration, page the on-call, re-run, close the incident when everything is green again. This is not naive at all as far as it goes: it is exactly right for the failures it can see, it costs nothing to build because the orchestrator emits it for free, and in most organisations it is genuinely the first monitoring worth having.

Why it breaks

The change-capture connector was down for part of the day. Every task processed everything it received, perfectly, and the records committed during the outage were emitted to nobody. There is no error anywhere because absence produces no signal (CDC Failure Modes and the Retention Deadline, Missing Rows).

How it breaks with real data

The change-capture connector was down for part of the day. Every task processed everything it received, perfectly, and the records committed during the outage were emitted to nobody. There is no error anywhere because absence produces no signal (CDC Failure Modes and the Retention Deadline, Missing Rows).
The broker redelivered part of the stream after a consumer restart. Nothing was lost, every task succeeded, and every additive measure is now inflated — which is questioned far less often than a shortfall (Duplicate Rows, Deduplication).
The producer started sending amount as a string. The cast produced null instead of raising, so the row counts match exactly, every task is green, and the measure is empty (Breaking Schema Changes, Nullability & Defaults).
The revenue model stopped subtracting refunds after an innocuous-looking edit. Every row is present, unique, fresh, well-typed and normally shaped. The only thing wrong with the output is the number (Two Dashboards, Two Numbers).
A historical re-run appended into the serving table instead of replacing the partition, so a period that was already correct is now present twice. The re-run was the *repair*, and it is what broke it (What Backfills Break, Upserts and Merges).
One key grew to hold most of the day. Nothing is wrong with the data at all — the job simply takes as long as its slowest task, and every task still reports success (Data Skew, Salting a Skewed Key).

Source→Ingestion→Raw→Transformation→Validation→Storage Model→Serving→Consumer→Observability

What is actually happening

A task status is the exit code of a process, evaluated against a timeout. It can only report failures that raise. Every failure class in this domain that matters — missing, duplicated, late, mistyped, or semantically wrong — is a valid computation over an input that was already wrong, so by construction none of them raises (Orchestration).
The pipeline is a sequence of set transformations, and each stage narrows or widens the set. A stage that receives a short set produces a short set and succeeds. Success propagates; completeness does not, because no stage compares what it received with what should have existed (Reconciliation).
This is testable rather than rhetorical. The model at src/de/sim/pipeline.ts pushes a fixed set of orders through source, change capture, broker, raw, transform, warehouse and dashboard, and reports the number the dashboard shows next to the number that is true. With no faults the two are equal exactly — not approximately — which is what makes the faulty runs interpretable at all.
Six checks run against that model — completeness, uniqueness, freshness, validity, distribution and reconciliation — and eight faults are injected one at a time. scripts/de-sim.test.ts asserts that no two faults trip the same set of checks. That property is what turns a list of symptoms into a diagnosis: the failing set identifies the fault (Data Tests).
The same suite pins the sharpest case. A broken transform trips reconciliation and nothing else, while every job stays green and every other check passes. If your platform has the other five checks and not that one, that fault is invisible to you by design (Data Quality).
Reconciliation is load-bearing because it is the only check that reads the source. Everything else can be computed inside the warehouse, which means everything else is a statement about the copy rather than about the world (Quality Alerting).

What a green run actually asserts

Here is a run report from a night when revenue was overstated. Nothing in it is fabricated for effect — this is what the orchestrator has to say, and it is a complete and honest account of what it observed. Every task started, every task finished, no task retried, and the durations are unremarkable.

Underneath it is the report nobody generated: what happened to the rows. The row counts are not merely plausible, they are *identical* at every hop, because the fault was not a loss. Refunded orders were counted at full value, which changes no count anywhere. The only artefact in the entire platform capable of noticing is a comparison against the source, and the platform did not make one.

Read the two blocks as answers to two different questions. The first answers "did the code run", which is a question about processes and is answered correctly. The second answers "is this period right", which is a question about records and has not been asked. The characteristic mistake of a data platform is to accept the first answer as a reply to the second (Where Did This Number Come From?).

Monitoring that the run happened

Alert when a task fails, when a run exceeds its expected duration, or when a retry budget is exhausted. Treat a green board as the definition of a healthy platform, and close incidents when the board is green again.

Monitoring that the period is right

Evaluate a named set of assertions against the published data for each period — completeness against the source, uniqueness on the declared key, freshness of the newest complete period, validity of the measures, distribution against history, and reconciliation of a summed measure — and treat the *set* that failed as the signal. Task status is one input among several.

Task status can only report failures that raise an exception. The failures that dominate this domain are valid computations over inputs that were already wrong, so they exit zero by construction. A monitor watching task status is not merely incomplete for them; it is structurally unable to observe them, and adding more task-level monitoring cannot change that.

RUN 2026-03-18T02:00Z                     status    duration   retries
  extract_orders                          SUCCESS   00:04:11        0
  land_raw                                SUCCESS   00:01:52        0
  stg_orders                              SUCCESS   00:02:38        0
  dim_customer                            SUCCESS   00:00:47        0
  fct_orders                              SUCCESS   00:06:02        0
  marts.revenue_daily                     SUCCESS   00:01:19        0
  refresh_dashboard_cache                 SUCCESS   00:00:12        0
                                          ALL GREEN

WHAT THE ROWS DID                        rows in   rows out   note
  source orders (period)                        -      4,000
  change capture                            4,000      4,000
  raw landing                               4,000      4,000
  transform (dedupe + window)               4,000      4,000
  serving table                             4,000      4,000
  dashboard                                 4,000          1   one number
                                     counts agree at every hop

  reconciliation vs source                              FAIL   revenue overstated

Eight faults, six checks, eight distinct fingerprints

SIMULATEDThese fingerprints are produced by src/de/sim/pipeline.ts with one fault injected at a time, and scripts/de-sim.test.ts asserts that no two of them coincide. A production platform differs in two ways that matter: faults arrive together and their fingerprints union, and real checks have thresholds, so a small enough instance of any of these rows trips nothing at all.

If diagnosis is going to be possible at all, different faults have to leave different traces. That is not automatic — it is a property a check portfolio either has or does not — and it is the property the in-repo model was built to demonstrate. Each fault below is injected alone, and the third column is the complete set of checks it trips.

Two things in the table are worth more than the rest. First, reconciliation appears in seven of the eight rows and is alone in one of them — it is both the broadest detector and the only detector of a transform that produces a complete, unique, fresh, well-typed and wrong result. Second, no two rows are identical, which is what makes the fourth column answerable at all.

Notice how close two of the rows come. A capture outage and a tail of late events both lose records and both trip completeness and reconciliation; they are told apart by distribution, because the outage removes a contiguous block large enough to change the shape of the day while lateness removes a thin tail that does not. Diagnosing by the first failing check would confuse them, and the two recoveries — replay a log gap versus widen lateness and reprocess — are different work (Late-Arriving Data, CDC Failure Modes and the Retention Deadline).

And notice the row where nothing about the data is wrong at all. Skew trips distribution alone: the numbers reconcile exactly, every record is present and unique, and the only consequence is that one task decides the runtime of the job. A check portfolio that treats every failing check as a correctness incident will page for this one and learn the wrong lesson from it (Data Skew).

Fault injected	What the operator sees	Complete set of checks tripped	What only this fault does
Change capture down for part of the day	All tasks green. The day looks quiet. Hop-to-hop counts agree perfectly.	completeness, distribution, reconciliation	Records never entered the pipeline, so every internal count is self-consistent. Only a comparison against the source can see it (CDC Failure Modes and the Retention Deadline).
Broker redelivers after a consumer restart	All tasks green. An additive measure is higher than expected. Nobody escalates.	uniqueness, reconciliation	Volume and shape stay close enough to normal that only a key-level assertion notices. Nothing was lost — this is an addition (Deduplication).
A share of events arrive after the window closed	All tasks green. The recent period looks slightly low and fills in if reprocessed.	completeness, reconciliation	The loss is a thin tail rather than a block, so the shape of the day is preserved and distribution stays green (Late-Arriving Data).
Producer sends a numeric field as a string	All tasks green. Row counts are exactly normal. The measure is empty.	validity, reconciliation	The only fault that preserves every count while destroying every value. Volume, freshness and uniqueness are all satisfied (Breaking Schema Changes).
The transform stops subtracting refunds	All tasks green. Everything looks completely normal. The number is too high.	reconciliation	The thesis case: it trips exactly one check. Present, unique, fresh, well-typed, normally shaped — and wrong (Two Dashboards, Two Numbers).
The transform raises and the run is abandoned	A red task. The dashboard renders yesterday with total confidence.	completeness, freshness, distribution, reconciliation	The only loud one, and the only fault that trips freshness — because nothing was published, not because anything was slow (Stale Dashboards).
One key grows to hold most of the day	All tasks green. The job takes much longer. One task is still running when the rest finish.	distribution	The only fault where the data is entirely correct. Adding workers does not help, because the straggler is one task (Data Skew, Salting a Skewed Key).
A re-run appends instead of replacing the partition	All tasks green. A period that was already correct is now present twice.	uniqueness, distribution, reconciliation	The fault introduced by the repair. It differs from redelivery by moving the shape as well, because a whole period was doubled rather than a sample of it (What Backfills Break).

Read a row right to left during an incident: you observe a failing set, and the set names the fault. Read the table top to bottom when designing a portfolio: a check missing from your platform makes every row that depends on it ambiguous, and dropping reconciliation collapses three rows into one.

The fault that trips exactly one check

The transform bug deserves its own section because it is the shape of every argument in this domain. The code did precisely what it was told. The data is complete: every order in the source is present. It is unique: each order id appears once. It is fresh: the run published on schedule. It is valid: every amount is a number. Its distribution is normal: the day looks like the days before it. And revenue is overstated, because refunded orders were counted at their full value.

Five of the six checks pass because five of them examine the copy. Only reconciliation looks outside, and only by comparing a summed measure — a comparison on row counts alone would also pass, since no row was added or lost. That is the specific reason reconciliation must compare a *measure* and not merely a count, and it is the most common way a reconciliation is built too weakly to catch the fault it was built for.

The table below is the model's own six checks, with what each expresses and what each still cannot see. Read the misses column as a set: its union is the space of failures your platform has decided to hear about from a human. That decision is fine, and it should be a decision (The Dimensions of Data Quality).

The six checks, and the blind spot each one keeps

Check	Expresses	Catches	Still misses
Completeness: every source record for the period is present	Nothing that happened was lost between the source and the serving table.	Capture gaps, dropped partitions, extract windows that closed early, filters that were too broad, late arrivals in a closed period.	Duplicates that coincidentally offset losses, and any period that is not yet closed — where a late record and a lost one are the same observation (Late-Arriving Data).
Uniqueness: each business key appears once	One row of this dataset really is one of the thing it claims to be.	At-least-once redelivery, a non-idempotent re-run, a fan-out join against a dimension with duplicate keys.	A duplicate that arrived under a new key. A producer retry with a fresh event id is two keys and one real event, and this check will confirm both are unique (Duplicate Rows).
Freshness: the newest complete period is recent enough	The pipeline has published a finished period within the window the consumer needs.	Crashed runs, paused schedules, stalled marts, a source that stopped producing entirely.	Data that is perfectly fresh and completely wrong, which is six of the eight faults. It also fires falsely on a period where the source genuinely produced nothing (Freshness Checks).
Validity: every measure is non-null and parses	The values are of the type the model believes they are.	A type change upstream, a cast that yields null rather than raising, a unit or format change that fails to parse (Nullability & Defaults).	A value that is well-typed and wrong. An amount in the wrong currency, a timestamp in the wrong zone and a quantity off by a factor all pass every type check there is.
Distribution: the shape resembles recent history	Today looks like a normal day, per dimension and in total.	Large contiguous losses, doubled periods, a key that has become dominant, a category that vanished at a deployment boundary (Distribution Tests).	Slow drift, and any error that preserves the shape while changing every value inside it — which is exactly what the transform bug does.
Reconciliation: a summed measure matches the source	The end of the chain agrees with the beginning about how much happened.	Every fault in the table except skew, including the one no other check sees (Reconciliation).	Anything wrong identically at both ends — shared logic between the extract and the model reconciles perfectly — open periods, and every column it does not sum.

Five of these six can be computed without leaving the warehouse. The sixth cannot, and it is the one that catches the fault the other five are blind to. That asymmetry is the practical content of this lesson.

A reconciliation that would have caught it, and one that would not

1-- Too weak: counts only. The transform bug adds no rows and loses none,
2-- so this passes while revenue is overstated.
3SELECT (SELECT COUNT(*) FROM src.orders     WHERE order_day = DATE '2026-03-18') AS source_rows,
4       (SELECT COUNT(*) FROM marts.fct_orders WHERE order_day = DATE '2026-03-18') AS served_rows;
5 
6-- Strong enough: compare a summed measure computed the same way at both ends,
7-- for a CLOSED period, and fail on any non-zero drift.
8WITH source AS (
9  SELECT SUM(CASE WHEN o.refunded THEN 0 ELSE o.amount_minor END) AS amount,
10         COUNT(*)                                                 AS rows_
11  FROM   src.orders o
12  WHERE  o.order_day = DATE '2026-03-18'
13), served AS (
14  SELECT SUM(f.net_revenue_minor) AS amount, COUNT(*) AS rows_
15  FROM   marts.fct_orders f
16  WHERE  f.order_day = DATE '2026-03-18'
17)
18SELECT source.rows_   AS source_rows,
19       served.rows_   AS served_rows,
20       source.amount  AS source_amount,
21       served.amount  AS served_amount,
22       served.amount - source.amount AS drift
23FROM   source, served
24WHERE  served.amount <> source.amount     -- must return zero rows
25   OR  served.rows_  <> source.rows_;

Two properties make the second query work and are both easy to lose. It sums a measure rather than counting rows, so a fault that changes values without changing counts is visible. And the refund rule is expressed independently on the source side rather than by calling the same model — a reconciliation that reuses the logic it is checking will agree with itself perfectly, forever.

Making green mean something

The conclusion is not that orchestrator monitoring is worthless — it is that it answers a different question, and the platform needs both answers published side by side. What remains is a design decision that is usually made by default and should be made explicitly: what happens when a check fails.

The honest framing is a trade between two costs. Publishing a wrong number costs whatever the consumer does with it, which is unbounded and invisible. Blocking a publish costs a stale dataset and a page, which is bounded and loud. The right answer differs per dataset, and the discriminator is what the consumer would do with a number that is late versus one that is wrong (Who Actually Consumes This Data).

The costs below are relative and directional, and they establish an ordering rather than a budget. What the ordering says is that the expensive part of a check portfolio is rarely its compute — it is the source-system capacity that the one genuinely valuable check consumes, and the human attention that the cheap checks spend when they fire without meaning (What Actually Drives Data Platform Cost).

What it costs to know, relative to each other

Reconciliation scans of the source system

The only check that spends the operational system's capacity rather than the warehouse's, which is why it needs the source owner's agreement and why it is the one most often dropped.

Recomputation for reproduction and anti-join checks

A second full computation of a closed period. Scales with history examined rather than with new data, so a naive schedule grows without anyone deciding it should.

Distribution comparisons against retained history

Cheap per run and dependent on keeping enough history to have a baseline at all — which is a retention decision made long before the check exists.

Row counts and freshness per hop

Nearly free, because the numbers are already produced by the run. Their real cost is the alerts they raise rather than the compute they use.

Human attention consumed by checks that fire without meaning

Not a compute cost and usually the binding one. A portfolio tuned until nothing fires has the same coverage as no portfolio and a much better reputation.

Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.

The ordering is the teaching: the most valuable check is the most expensive and the most political, and the cheapest checks are limited by attention rather than by compute. A platform that optimises this chart by dropping the top row has optimised away the only check that sees the fault in the previous section.

What should a failing check do?

For this dataset, is a stale number worse than a wrong one — and who finds out either way?

Warn, publish anyway

when The check has a known false-positive rate, the consumer can tolerate an anomaly, and someone reliably reads the warning within the period that matters.

cost The wrong number reaches the consumer first. Every warning that turns out to be noise makes the next real one less likely to be read, and the failure mode is a channel nobody watches (Alert Fatigue: The Page Nobody Reads).

Block the publish

when A wrong number is worse than no number — finance, external reporting, anything that drives an irreversible decision or leaves the company.

cost A silent data error becomes a visible outage and a page, including for faults the consumer would not have noticed. That is usually the right trade and it is not free (Contract Enforcement).

Quarantine the failing subset

when The fault is confined to an identifiable slice — one source, one region, one client version — and the rest of the period is genuinely usable.

cost Consumers now read a period that is knowingly partial, which is only safe if the partiality is published with it. An unannounced quarantine is a silent shortfall you created deliberately (Missing Rows).

Publish, then revise

when Lateness is normal and the consumer understands provisional figures — most streaming-fed dashboards and any period that is still open.

cost Numbers change after they are quoted. This is correct behaviour and it destroys trust unless the provisional status is visible on the tile itself (Late-Arriving Data).

Roll back to the last good publish

when The serving table has a previous version you can restore atomically and the previous period is more useful than a gap.

cost You are now serving data you know to be stale while presenting it as current, so the freshness signal has to move with the rollback or the platform is lying twice (Rolling Back Data).

How to build it

Most important first.

Publish a data status alongside the pipeline status, per serving dataset: freshness against its stated target, row count against its own history, uniqueness on the declared key, and reconciliation for the last closed period. Four facts, visible to the consumer, none of them derived from a task exit code (The Data Quality Dashboard, Data Observability).
Build reconciliation first, before the cheaper checks, even though it is the most expensive and the only one that needs source access. It is the check with the broadest coverage, and it is the one a platform is most likely to skip precisely because it crosses a system boundary (Reconciliation).
Decide, per dataset, what a failing check does. Warning is right for a signal with a known false-positive rate; blocking the publish is right where a wrong number is worse than no number; quarantining the batch is right where the rest of the run is still useful (Contract Enforcement).
Make the publish atomic so that "the run failed" and "the consumer saw half a dataset" cannot both be true. A partial publish is the case where a loud failure still produces a silent data error (Atomic Publish, Partial Failure).
Assert the expected period, not just the presence of rows. A run that publishes on time and publishes nothing satisfies every schedule-based monitor there is (Volume Anomalies).
Record the failing-check set on every incident. Over a year that set becomes the fingerprint table for your own platform, which is worth more than any generic one, including this lesson's (Data Incidents).

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

A successful task guarantees that a process started, ran to completion within its timeout and returned zero. That is a real guarantee about code, it is honoured, and this lesson is not an argument against having it.
It guarantees nothing about completeness. A stage that received a set missing four hours of changes processes it successfully and passes it on (Missing Rows).
It guarantees nothing about uniqueness. At-least-once delivery is normal behaviour rather than a malfunction, and a duplicate is an addition, which no stage has a reason to reject (Duplicate Rows).
It guarantees nothing about meaning. A transformation that faithfully implements a wrong definition is a correct program, and no technical check will ever say otherwise (Semantic Changes).
What a check guarantees is narrower than people assume: that its specific assertion held on the data it examined, over the period it examined. Its blind spot is as much a property of your platform as its coverage.

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this

The check that answers this lesson is reconciliation against the source for a closed period, on both a count and a summed measure. It is the only one of the six that can see a fault which occurred before your pipeline, and the only one that catches a transform whose output is complete, unique, fresh and wrong.
It misses anything wrong identically at both ends — a bug in logic shared by the extract and the model reconciles perfectly — every period that is still open, where lateness and loss are indistinguishable, and every column it does not sum.
Which is why the answer is a portfolio rather than a check. Run all six, know the blind spot of each, and state deliberately which failures you have decided to learn about from a consumer (The Dimensions of Data Quality, Who Owns Data Quality).

Freshness

A green run tells you a process finished; it does not tell you the data is current. The two diverge in both directions — a run that succeeds having published nothing leaves yesterday on the dashboard, and a run that fails loudly leaves the previous period in place looking exactly as confident as it did before (Stale Dashboards).
In the model, the crashed transform is the one fault that trips freshness, and it does so because nothing was published rather than because anything was slow. Freshness is a check on publication, not on latency (Freshness Checks).
Every other fault in the set leaves freshness perfectly green while the number is wrong, which is why a freshness monitor alone produces the most confident wrong impression of health available (The Freshness SLO).

When the schema or meaning changes

Schema changes are the fault class most likely to pass every check you own, because a type change that casts to null preserves row counts and therefore preserves volume, freshness and uniqueness. Validity is the check that sees it, and validity is usually the last one anyone writes (Schema Evolution).
When a contract check is added, a formerly silent fault becomes a loud outage. That is the correct trade and it should be made explicitly, because the first time it fires somebody will ask why the pipeline is now less reliable than it used to be (Data Contracts).
Adding a check changes what "green" means, so the historical record of green runs is not comparable across the change. Version the check set the way you version the code (Data Tests).

How to re-run this safely

Diagnose before you re-run. A re-run of a deterministic pipeline over an input that was already short reproduces the shortfall exactly and consumes the incident's first hour proving that the code works (Idempotent Data Pipelines).
Recovery differs per fault and the fingerprint tells you which one you have: replay from the retained log for a capture gap, deduplicate on the producer's key for redelivery, widen lateness and reprocess for late arrivals, fix the logic and backfill the affected range for a transform bug (Replay from the Log, Planning a Backfill).
Fixing the code forward does nothing about the history already published. Those are two pieces of work with different risks, and shipping only the first leaves a permanent notch that somebody finds a year later (Validating a Backfill Before You Publish).

What can go wrong

Failure modes

The platform has five of the six checks and not reconciliation, so the entire class of "complete, unique, fresh and wrong" is structurally invisible.
A check is written against the serving table on both sides of the comparison, so it proves that copying is deterministic and nothing else.
Checks run *after* publish and only alert, so the wrong number reaches the consumer first and the alert arrives to an audience that has already acted on it.
The distribution check is widened every time it fires until it cannot fire, which is how a platform arrives at four checks it believes are five (Alert Fatigue: The Page Nobody Reads).
A fault is diagnosed by the first failing check rather than by the failing set, so a capture gap and a late-arrival tail — which share two of their three checks — are confused, and the wrong recovery is run.
The check suite itself fails to run, and its absence is reported as an absence of failures.

Misreads

"All the tasks are green, so the data is correct." This is the belief the domain exists to break, and it survives because it is right the overwhelming majority of the time — which is exactly what makes the exceptions expensive.
"The run failed, so nothing was published." Only if the publish is atomic. A partial write is a loud failure and a silent data error at the same time.
"We have data tests." Ask which of the six properties they cover and what each one misses. Five tests that all read the warehouse cover one side of the comparison.
"Re-run it and see." A deterministic pipeline over an unchanged input returns the same wrong answer, and the re-run has told you only that the code is a function.
"The check passed, so that fault did not happen." A check passing means its assertion held over what it examined. Reconciliation over an open period cannot distinguish a late record from a lost one, and it passes either way.
"The first failing check names the fault." Faults share checks. Only the failing *set* is distinctive, which is the entire point of the fingerprint table.

Operating it

How you see it in production

The failing-check set per run, kept as a set rather than as a first alert. The set is the diagnosis; any single element of it is a symptom (Pipeline Observability).
Row counts per hop, per run, on one chart, so a loss is localised to an arrow instead of searched for across six systems (Pipeline Metrics).
Reconciliation drift over time as a value rather than a boolean. A drift that is zero every day and then small is far more informative than a check that flips from pass to fail (Reconciliation).
Publish outcomes as a first-class event: which period, which run identifier, how many rows, which checks were evaluated and what each returned (Debugging a Data Incident).

What changes at 10x and 100x

At 10x datasets, the check portfolio has to become a property of a dataset rather than a set of hand-written tests, or coverage becomes a function of which team was diligent (Data Tests).
At 100x volume, reconciliation over a full period stops being affordable and becomes sampled or bucketed — by day, by region, by key range — which weakens it in a specific and stateable way rather than uniformly (Reconciliation).
Consumer count changes what a failing check has to do. With three consumers you can tell them; with eighty, the check has to block the publish, because you cannot reach the people who already acted on the number (Who Actually Consumes This Data).

What drives cost here

Reconciliation costs a scan of the source for the compared period, which is the only check that spends the operational system's capacity rather than the warehouse's. That is precisely why it is the one platforms skip, and why the argument for it has to be made to whoever owns the source (Scan Cost).
The cheap checks — volume, freshness, per-hop counts — cost close to nothing because the numbers are already being produced. Their cost is not compute; it is the alerts they raise and the attention those consume (Alert Fatigue: The Page Nobody Reads).
The expensive checks are the ones that recompute: reproduction of a closed period, anti-joins against source keys, and distribution comparisons over history. Run those on a schedule and on the datasets whose numbers leave the company, not on everything (Compute Waste).

What this approach costs

Every check added is another thing that fires, another thing to tune and another thing that can be wrong. A platform with fifty tests nobody trusts is worse off than one with five that block a publish, because the fifty consume exactly the attention the five needed.
Blocking a publish converts a silent wrong number into a visible outage. It is usually the right trade and it will still page somebody at an unreasonable hour for a fault whose consumer would not have noticed until Monday.
Reconciliation is the highest-value check and the one that couples you to the source system's availability and capacity. Buying the coverage means accepting that a source under load can now fail your data pipeline's quality gate.

Diagnose it from the signals

Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.

Diagnose it from the signals

Only the check results are shown. One fault produced them — and exactly one can have, because no two faults in this model fail the same set.

this morning's check results

Check	Result
Completeness Every order the source recorded for the period reached the serving table.	FAIL
Uniqueness Each order id appears exactly once in the serving table.	pass
Freshness The newest complete record is recent enough for the decisions this table drives.	pass
Validity Every amount is non-null and parses as a number.	pass
Distribution The shape of the day resembles the days before it, per country and in total.	FAIL
Reconciliation Revenue summed in the serving table equals revenue summed in the source for the same closed period.	FAIL

which fault produced this?

After you answer, every fault's fingerprint is shown. Read them as a set: two faults that both fail reconciliation are told apart by what else moved, not by that.

0/0 correct · 1 of 8SIMULATEDFingerprints are computed by running each fault through the pipeline model.

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

GENERALThat task success reports on process execution rather than on data correctness is true of every orchestrator ever built, because an exit code is the only thing a process can return. What differs between stacks is how much data-level signal the tool happens to emit alongside it, not whether the gap exists.
SIMULATEDThe eight faults, six checks and their fingerprints come from the model at src/de/sim/pipeline.ts and are asserted by scripts/de-sim.test.ts. They are properties of a deliberately small teaching model, not measurements of a production platform: a real platform has more faults, coarser checks, and faults that occur together rather than one at a time.
TOOL-SPECIFICSome orchestrators can express a data assertion as a task, so a failing check becomes a failing task and the distinction blurs usefully. That is a property of how you wired it rather than of the tool: the same tool run with checks in a separate system after publish gives you green tasks and wrong data again.

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

Databasetransactions-and-acid aggregation-and-grouping follow-the-query

Backendscheduled-jobs job-idempotency health-checks at-least-once-delivery

API Designapi-testing

Observabilitysymptom-to-signal golden-signals alerting alert-fatigue incident-debugging

Cloudliveness-vs-readiness

Domains that do not exist yet

— Distributed Systems owns why a stage cannot know what it did not receive: absence produces no message, and no amount of local success can establish a global property. When that domain lands, the completeness check here should link into it.
— DevOps / Production Engineering owns the delivery half — how a check suite is versioned and promoted with the transformation code, and why a gate that can be skipped under deadline pressure is a gate that will be.