FoundationsGENERALSIMPLIFIEDSCALE-SPECIFIC
What Goes Wrong Between Source and Dashboard
Thirteen failure classes, each with its own mechanism, its own detector, and a long list of checks that will never find it.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
The question
The job succeeded, the table has rows, and the number is wrong — which of the thirteen things that go wrong is this one?
Who needs this
Whoever is holding the incident. An analyst who noticed the trend broke, a finance controller who cannot close the month, an on-call engineer with a green orchestrator and an angry Slack thread. All three need the same thing: a short list of candidate mechanisms and a cheap test that eliminates most of them.
What one row is
One row of this taxonomy is one mechanism, not one symptom. That is the unit deliberately: several mechanisms produce identical symptoms (a low number can be missing rows, a bad cast, a partial load or a filter change), and several symptoms come from one mechanism. Detection is designed per mechanism, so the taxonomy has to be cut that way too.
The obvious build
Treat data incidents as one category — "the pipeline broke" — and debug each one from first principles: open the DAG, look at the failed task, find there isn't one, then start reading SQL. This works, slowly, and it works best on the incidents that were already the easiest.
Why it breaks
Every incident costs a full investigation, because nothing learned from the last one narrows this one. Teams that have run twenty data incidents without a taxonomy are no faster on the twenty-first.
How it breaks with real data
- Every incident costs a full investigation, because nothing learned from the last one narrows this one. Teams that have run twenty data incidents without a taxonomy are no faster on the twenty-first.
- The checks that get written are the ones that would have caught the *previous* incident, one at a time, forever. The portfolio ends up dense in one corner and empty everywhere else (The Dimensions of Data Quality).
- Failures with no loud symptom are never investigated at all, because nobody reported them. Out-of-order updates and semantic drift can run for quarters and are usually found by accident (Semantic Changes).
- The response is chosen before the mechanism is known. "Re-run it" fixes a transient failure, does nothing for a schema change, and actively makes duplication worse (Idempotent Data Pipelines).
- The same wrong number gets explained three different ways by three different people, and the one who sounds most confident wins.
Source→Ingestion→Raw→Transformation→Validation→Storage Model→Serving→Consumer→Observability
What is actually happening
- Every failure here is a change to the record in transit — it does not arrive, it arrives more than once, it arrives at the wrong time, it arrives with the wrong type, or it arrives intact and is then transformed into something that does not mean what its name says.
- That gives four families. Loss and duplication are about count. Timing — late, out-of-order, stale, partial — is about which records are present at the moment someone reads. Meaning — schema, types, transformation, business logic — is about records that are all present and all wrong. Machinery — backfills, skew, job failure — is about the pipeline itself, and reaches the data indirectly.
- Detection sorts along the same lines and does not cross them. A count-based check finds count problems and is blind to meaning. A schema check finds shape problems and is blind to semantics. A freshness check finds absence and is blind to everything present (Data Tests).
- The single most important structural fact: eleven of the thirteen leave the pipeline green. Only outright job failure and, sometimes, skew produce a signal the orchestrator can see. Everything else runs to completion (The Pipeline Succeeded. The Data Is Wrong.).
- The second structural fact: the detector for staleness cannot live inside the pipeline. If the run did not happen, neither did any test it contained. That is why freshness monitoring is an external observer and not a data test (Freshness Monitoring).
The thirteen, and what actually detects each one
GENERALThe detector column names a category of observation rather than a product: "reconciliation" is a scheduled query in any warehouse, and the row holds whether it runs in dbt, a notebook or a cron job. What changes per stack is cost, not applicability.
Read the third and fourth columns together. The value of a taxonomy is not that it names things; it is that each row pairs a mechanism with the one cheap observation that reveals it, and with the observations that will never reveal it no matter how many of them you run.
The fourth column is the one worth memorising. Almost every false sense of safety in a data platform comes from a check that is genuinely good at its own class being trusted for a neighbouring one — counts trusted for values, schema trusted for meaning, freshness trusted for correctness.
Note how often "the orchestrator" is absent from the third column. It appears once. That single fact is why data observability exists as a discipline separate from pipeline monitoring (Data Observability).
| Failure class | What it looks like | What detects it | What never will |
|---|---|---|---|
| Missing rows | A period is quietly low; the trend has a step in it that nobody changed. | Reconciliation of counts against the source for a closed period. | Task status, uniqueness tests, freshness — all pass with rows missing. |
| Duplicate events | A metric is high; a count of distinct business keys is lower than a count of rows. | Uniqueness assertion on the business key, at the grain the table declares. | Row-count reconciliation alone, if duplication and loss coincide and offset. |
| Schema changes | A column is null everywhere, or a model fails on a name that no longer exists. | Contract or registry compatibility check at the ingestion boundary. | Any test written against the columns that still exist. |
| Late data | A closed period keeps changing; yesterday's number is different today. | Comparing a published event-time partition against itself on later runs. | Freshness — late data is fresh on arrival; it just belongs to the past. |
| Out-of-order events | One key holds a stale value while its own newer change sits in the raw layer. | Asserting the selected record carries the maximum source sequence for its key. | Uniqueness and counts: there is exactly one row per key, and it is the wrong one. |
| Bad types | A sum drops while the count holds; a timestamp lands in the wrong day. | Null-rate per column against its history, plus explicit type assertions at the boundary. | Row counts — the row is present, it simply carries null where a number belonged. |
| Broken transformation | The fact table has more rows than the model it derives from. | A grain assertion after every join, plus row count against the parent model. | Schema tests, null tests, freshness — the output is well-formed and wrong. |
| Stale data | A confident number from last Tuesday, rendered exactly like a current one. | A freshness observer running *outside* the pipeline against a stated SLO. | Every in-pipeline test: if the run did not happen, the tests did not run either. |
| Partial loads | A period looks merely low rather than incomplete; a re-read gives a different total. | Atomic publish plus an explicit completeness marker per period. | Freshness — the newest partition is recent even when older ones are absent. |
| Backfills | History changed, or doubled, or the current partition was overwritten by an old one. | Snapshot the range before, diff after, validate in a staging location before publish. | Any check scoped to the current partition, which is where checks are usually scoped. |
| Partition skew | One task runs for hours; the job misses its window and the dataset goes stale. | Per-task input distribution — the ratio of the largest partition to the median. | Data tests: every row is correct, and correctness is not the failing property. |
| Pipeline failure | The loud one. A red task and a page. | The orchestrator, which is genuinely good at this and only this. | Nothing — but watch for what a partially-completed retry left behind. |
| Incorrect business logic | The number is precise, stable, reproducible and does not mean what its name says. | A consumer who knows the domain, or reconciliation against an independently produced figure. | Every technical check, by construction — the code did exactly what it was told. |
Count and timing: the six that change which rows are there
These six share a property that makes them tractable: they are all visible in counts, if you count the right thing over the right window. They are also the six most likely to be blamed on each other, because a low number is a low number regardless of why.
The trigger column is the part worth studying. Each of these has a specific, boring, repeatable cause — a predicate against a timestamp assigned before commit, an offset committed after a write, a partition closed on arrival time rather than event time. None of them is exotic, and all of them recur.
Notice that the response differs sharply even where the symptom is identical. Re-running is correct for a transient extract failure and actively harmful for a duplication incident, and the two present the same way in a dashboard until you count distinct keys.
- Five of these six leave the pipeline green. The sixth — a paused DAG — is green in a worse way: nothing failed, because nothing ran, so no test ran either.
- Three of them (missing rows, out-of-order events, partial loads) produce a symptom that is indistinguishable from the others without counting distinct keys or comparing against the source.
Loss, duplication and timing
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
A nightly extract runs WHERE updated_at > :high_water against a source that assigns updated_at when the statement runs, not when it commits. | A small, steady, permanent shortfall. Nobody notices until a reconciliation runs or an audit compares against the source. | A row whose transaction committed after the watermark advanced past its timestamp is never inside any future window. It is skipped forever, not delayed (Incremental Extraction). | Move the watermark to a commit-ordered position — a log sequence number or change stream position — and re-extract the affected range with an overlap (The High-Water Mark). |
| A consumer writes its output, then crashes before committing its offset. On restart it re-reads from the last committed position. | Revenue is high. Row count exceeds distinct business keys. The fact table gained rows nobody inserted twice on purpose. | At-least-once delivery is the normal, correct behaviour of every log and queue in this position. The duplicate is not a bug in the broker (Offsets and Commits). | Make the write idempotent — merge on the business key rather than append — and deduplicate the affected range. Fixing the writer without cleaning history leaves the wrong number in place (Upserts and Merges). |
| An event occurs at 23:58 and arrives at 00:04, after the previous day's partition has been computed and published. | Yesterday's number is correct in the morning and different by lunchtime, or never gets the event at all depending on how the partition was written. | The pipeline partitions by arrival time while the business question is about event time. The two agree most of the time, which is what makes the disagreement so hard to see (Event Time). | Partition by event time, define an allowed lateness explicitly, and reprocess the event-time partitions a late batch touches — not the ingestion window it arrived in (Late-Arriving Data). |
| Two changes to the same order are produced to different partitions, or retried out of sequence, and "latest per key" is computed by arrival order. | One customer's tier, one order's status, one address is stale — and only that one. There is exactly one row per key, so nothing looks wrong. | Ordering is a per-partition property in every log that offers it at all. A key whose partition assignment changed loses order relative to its own history (CDC Ordering and Transaction Boundaries). | Choose the latest by a source-assigned sequence — LSN, binlog position, version column — never by arrival. Then assert that the chosen row carries the maximum sequence for its key (Event Keys and Partition Assignment). |
| A transformation writes twelve partitions in twelve statements and fails after the seventh. | A period reads as complete and low. A consumer who re-reads an hour later gets a different total and assumes they mis-read the first time. | The publish was not atomic, so intermediate states are observable. Warehouses offer transactional writes; a pipeline that does not use them has as many observable states as it has statements (Atomic Publish). | Write to a staging location and swap in one operation, or use the table format's commit. Publish a completeness marker per period so consumers can tell "no data" from "not yet" (Partial Failure). |
| The upstream source stops producing, or the DAG is paused during a migration and nobody un-pauses it. | A dashboard renders a confident, precisely-formatted number that has not moved since Friday. Nothing is red. | Absence produces no event. Every in-pipeline check is subject to the same absence — the run that would have failed the freshness test also did not run (Freshness Checks). | Run the freshness observer outside the pipeline, on a schedule the pipeline cannot pause, comparing the newest complete record against the dataset's stated SLO (Freshness Monitoring). |
Meaning and machinery: the seven where every row is present
The remaining seven split into two kinds. Four are about the rows being wrong while all present — a schema change, a bad type, a broken transformation, a wrong definition. Three are about the machinery, and they reach the data indirectly, usually by turning into staleness or into corrupted history.
The last row of this table is the one worth arguing about. Incorrect business logic has no technical detector and never will, because there is no observable difference between a transformation that implements the wrong definition correctly and one that implements the right definition correctly. The only detector is a person who knows what the number is supposed to mean, and the only engineering response is to make the definition explicit and reviewable somewhere other than inside a SQL file (The Metrics Layer).
The backfill row deserves the same attention for the opposite reason: it is the only class where the *response to another incident* is itself the incident. More history has been destroyed by a well-intentioned recompute than by any source system (What Backfills Break).
- The last three rows are machinery failures. Each converts into a data failure from the earlier table — skew becomes staleness, a bad retry becomes duplication, a bad backfill becomes wrong history.
- The first four are meaning failures, and they are ordered by how detectable they are. Schema is mechanical, types are statistical, grain is assertable, and definition is none of the three.
Wrong values, wrong meaning, wrong machinery
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
An upstream team renames amount_cents to amount_minor in a correct, reviewed migration. | Revenue reports zero, or the column is null everywhere, depending on whether the model referenced it by name or selected everything. | Nobody knew analytics read that column, because nothing recorded the dependency. The migration was correct in its own domain (Breaking Schema Changes). | Enforce a contract at the boundary so the change is rejected loudly at ingestion rather than absorbed silently, and publish the dependency so the producer can see who reads it (Data Contracts). |
| A numeric field starts arriving quoted, or a timestamp arrives without a zone, and the load casts rather than raises. | A sum drops while the count holds steady. A day's events land in the wrong day for a subset of rows. | SUM skips nulls silently and COUNT(*) does not care about them, so a cast-to-null is invisible to the two most commonly written checks (Nullability & Defaults). | Assert types at the boundary and track null rate per column against its own history. A step change in null rate is the cheapest value-level detector there is (Distribution Tests). |
| A dimension gains duplicate rows after a non-idempotent re-run, and a fact table joins to it. | Revenue is a clean multiple of itself. The job ran faster than usual, because nothing raised. | The join fanned out: one fact row matched several dimension rows, and every measure was multiplied by the match count (Grain: What Does One Row Represent?). | Assert uniqueness on every join key before the join, and assert output row count against the parent model after it. Both are one-line tests and both are usually missing (Data Tests). |
The revenue model filters status = 'complete' because that is what the column looked like; the business means "shipped and not refunded". | A number that is precise, stable, reproducible, defended in review, and wrong. It may be wrong for years. | The definition lived in a SQL file, was never written down in business terms, and was never reviewed by anyone who owned the metric (Two Dashboards, Two Numbers). | Move definitions into a reviewable metrics layer, and reconcile against an independently produced figure — finance's ledger is the usual one (The Metrics Layer). |
| A six-month backfill for a logic fix is launched against the production table while consumers are reading it. | History changed under a report that had already been circulated; or the current partition was overwritten with recomputed values from an older source state. | The backfill was treated as "re-running the DAG with different dates". It is a migration of history and needs the same care (Backfills). | Bound the range explicitly, write to a location nobody reads, validate against the pre-change snapshot, then publish atomically (Validating a Backfill Before You Publish). |
| One tenant, one country or one null key holds most of the rows, and the shuffle sends them all to one task. | The job takes hours instead of its usual window and eventually misses it. The data, when it arrives, is correct. | Work is distributed by key hash, so a key distribution with a heavy head produces a task distribution with a heavy head (Data Skew). | Measure the ratio of the largest partition to the median before optimising anything, then salt, broadcast or repartition as the distribution warrants (Salting a Skewed Key). |
| A task fails outright — a bad credential, an out-of-memory kill, an upstream timeout. | A red task and a page. The one failure class the existing monitoring was built for. | Something raised. This is the easy case, and it is worth listing precisely because it is the one everybody already handles (When a Task Fails Mid-DAG). | Retry only if the step is idempotent. A retry of a non-idempotent write turns a loud, simple failure into a quiet duplication incident (Retries in Pipelines). |
Building the portfolio: what each check buys and what it leaves open
Once the classes have detectors, the practical question is which detectors to actually run. The answer is not "all of them": each costs a scan, an alert budget and someone's attention, and a portfolio nobody trusts is worse than a smaller one everybody does.
Choose by consequence rather than by coverage. For each serving dataset, ask which of the thirteen would be *expensive* if it happened here — a duplicated dimension feeding an executive metric is a different risk from a null rate in a column no dashboard reads — and buy detectors in that order.
The misses column below is not a caveat, it is the design. Every check has a blind spot, the blind spots are what you are choosing between, and a portfolio is complete when its remaining blind spots are ones you have decided you can live with (The Data Quality Dashboard).
Six detectors, their class coverage, and their blind spots
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Reconciliation: source versus serving, closed period, count and a summed measure | The two ends of the journey agree about a period nobody will add to. | Missing rows, duplicates, fan-out joins, and a partial load that was never repaired. | Open periods entirely, which is where late and out-of-order data live; any error present identically at both ends; every column not summed. |
| Uniqueness on the declared business key | The table is at the grain it claims to be at. | Redelivery, non-idempotent re-runs, fan-out from a duplicated dimension. | Duplicates that differ in the key — one order re-emitted with a fresh event id is two orders to this check and one order to the business. |
| Freshness of the newest complete record, evaluated outside the pipeline | The data is recent enough for the decision it drives. | A stopped pipeline, a paused DAG, a source that went quiet, a job that missed its window because of skew. | Everything about data that is present. Fresh and wrong passes cleanly, and it fires falsely on periods where the source genuinely produced nothing. |
| Null rate and distinct count per column, against that column's own history | The shape of the values has not shifted. | Cast-to-null, a renamed upstream column absorbed by SELECT *, an enum that gained a value, a join key that stopped matching. | Any change that preserves shape — a unit change, a currency change, a sign flip, a definition change. All of these look perfectly normal. |
| Row count per hop, per run, compared against the adjacent hop | No arrow in the journey is losing or multiplying records. | Loss localised to one ingestion step; fan-out localised to one transformation; a partial load between two layers. | Losses that are compensated by duplicates in the same run; anything wrong at the source before the first hop. |
| Reconciliation against an independently produced figure — a finance ledger, a payment provider report | The number means what the business believes it means. | Incorrect business logic, semantic drift, a definition that diverged from the one an executive is quoting. | Anything the independent figure also gets wrong; and it is slow, manual and usually monthly, so it finds problems long after they started. |
Only the last row can find the thirteenth class, and it is the only detector on this list that is not a query. That asymmetry is permanent: meaning is not a property a machine can assert.
Product detail — verify current documentation
Test frameworks, observability products and warehouse-native constraint features all move quickly and each covers a different subset of these six. Evaluate any of them by asking which rows of this table it implements and which blind spots it leaves — not by feature count.
How to build it
Most important first.
- Learn the thirteen as a checklist and walk it during an incident instead of reasoning freely. Most incidents are eliminated to two or three candidates in under a minute by asking: is the count wrong, is the timing wrong, or is the meaning wrong?
- Assign each class a detector and write down what that detector misses. A check whose blind spot is unstated is worse than no check, because it converts "we do not know" into "we verified it" (Data Tests).
- Put the detectors where the failure happens. Completeness belongs at ingestion, grain and fan-out belong immediately after each join, semantics belong at the contract boundary, freshness belongs outside the pipeline entirely (Contract Enforcement).
- Reconcile against the source for closed periods on a schedule, not on demand. It is the only check that observes both ends of the journey at once, and it is the one nobody runs until the incident (Reconciliation).
- Record which class each incident turned out to be. After a dozen, the distribution tells you where your platform is actually weak, which is almost never where the team assumed (Data Incidents).
- Make the response class-specific and write it next to the detector. "Re-run" is correct for exactly one of the thirteen and dangerous for at least three (What Backfills Break).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- A green orchestrator guarantees that every task exited without raising, within its timeout. It says nothing about rows, values, timing or meaning, and treating it as evidence is the domain's defining mistake.
- A passing data test guarantees the assertion you wrote holds on the data it ran against. It does not guarantee the assertion is the right one, and it guarantees nothing about the rows a filter excluded before the test saw them.
- A reconciliation against the source guarantees the two ends agree on the aggregate you compared, for the period you compared, at the moment you compared it. Open periods and uncompared columns are outside the guarantee.
- Nothing in a normal pipeline guarantees that a value means the same thing this month as it did last month. There is no type for meaning (Semantic Changes).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
The check that would catch this
- The strongest single check is a scheduled reconciliation on a closed period: row count and a summed measure, source versus serving table. It covers missing rows, duplicates and fan-out joins with one query.
- It misses every open period — which is exactly where late and out-of-order data live — and it misses any error that is present identically at both ends, which includes every case where the bug is in shared logic.
- It is also silent on meaning. A column that changed from gross to net reconciles perfectly and reports the wrong thing, forever, with a passing check next to it (Two Dashboards, Two Numbers).
Freshness
- Three of the thirteen are purely about time — late data, out-of-order events and stale data — and none of them is visible in a snapshot. You only see them by comparing a dataset against itself at two moments, or against a clock (Late-Arriving Data).
- A partition that is "complete" is a claim about time, not about rows: it says no further records for this period will arrive. Almost no pipeline states when that becomes true, which is why consumers read open periods as if they were closed.
- Skew and job failure convert into staleness rather than into wrongness. The rows that eventually arrive are correct; they arrive after the decision was made.
When the schema or meaning changes
- Two of the thirteen are evolution failures by definition: a schema change that alters shape, and a semantic change that alters meaning while leaving the shape untouched. The second is strictly harder and has no automated detector (Semantic Changes).
- Adding a field upstream is usually safe and occasionally not — a
SELECT *staging model widens, and any downstream consumer with a positional or exhaustive assumption breaks (Backward Compatibility). - The taxonomy itself evolves as a platform grows. A team with no streaming has no out-of-order class; the day CDC lands it acquires one, and nobody notices until an update wins that should have lost (CDC Ordering and Transaction Boundaries).
How to re-run this safely
- Recovery is class-specific and this is the practical payoff of the taxonomy. Missing rows: re-extract the range and merge. Duplicates: deduplicate on the business key, then fix idempotency at the writer (Deduplication).
- Late and out-of-order data: reprocess the affected event-time partitions, not the ingestion window they arrived in — those are different ranges and confusing them is how a backfill misses its own target (Late-Arriving Data).
- Meaning failures cannot be recovered by re-running: the code has to change first, then history is recomputed from raw, then the two versions of the number have to be explained to whoever quoted the old one (Planning a Backfill).
- The one class with no recovery path is a source that has since been mutated or a raw partition that was overwritten. Everything else is a re-run; this is a loss (Keeping Raw History: The Recovery Position and the Liability).
What can go wrong
Failure modes
- The wrong class is diagnosed, the wrong fix is applied, and the fix creates a second incident on top of the first — the classic being a re-run against a non-idempotent writer, which turns missing rows into duplicated rows.
- A check is added that would have caught the last incident and is then trusted for classes it cannot see.
- A detector fires constantly for a benign reason (a genuinely quiet weekend, a source that has no data on holidays) and is muted, which removes it from the portfolio without removing it from anyone's mental model (Quality Alerting).
- Two classes co-occur — a schema change causes a cast to null, which drops a sum but not a count — and the investigation stops at the first plausible cause.
- The taxonomy is used as documentation rather than as a runbook, so nobody walks it under pressure.
Misreads
- "Most data incidents are pipeline failures." Pipeline failures are the ones you *hear about* immediately. The distribution of incidents that reached a consumer is completely different from the distribution of alerts.
- "If row counts match, the data is fine." Counts are invariant to almost every value-level failure — a bad cast, a wrong filter on a column you did not sum, a rename that nulled a dimension — all preserve count exactly.
- "We have data tests, so we have coverage." Coverage is per class, not per test count. Twenty uniqueness tests cover one of thirteen classes.
- "A low number means missing data." It is the most common cause and it is far from the only one. Bad casts, wrong filters and partial loads all produce low numbers, and a duplicated dimension produces a high one that nobody questions (Duplicate Rows).
Operating it
How you see it in production
- Row count per hop per run, plotted together. A drop between two adjacent hops localises loss to one arrow; a rise localises duplication or fan-out to one transformation.
- Freshness per serving dataset, measured by an observer outside the pipeline, with the dataset's stated SLO on the same axis (The Freshness SLO).
- Null rate and distinct count per column, per run. These two cheap series catch the type and cast failures that row counts cannot see (Distribution Tests).
- Reconciliation divergence as a time series rather than a boolean. A gap that grows slowly is a different incident from one that appeared overnight.
What changes at 10x and 100x
- At 10x volume the classes do not change, but the *detectors* get expensive and start being sampled or scoped, which quietly reintroduces blind spots.
- At 100x, skew stops being a performance concern and becomes a correctness one: a job that cannot finish inside its window produces staleness, and staleness is read as a number (Data Skew).
- Consumer count multiplies the blast radius rather than the failure rate. The same missing rows reach eighty dashboards instead of three, and the number of people who must be told a number was wrong is what actually costs the team (Impact Analysis).
What drives cost here
- Detection costs scans. A reconciliation reads a period at both ends; a distribution test reads a column across the run; a uniqueness test on a wide table can be the most expensive query in the platform (Scan Cost).
- The cost is controlled by scope, not by frequency: test the columns and periods that matter rather than everything, and let layout do the work of not reading what you did not ask for (Partition Pruning).
- Undetected failures cost more, but the cost lands in a different budget — a re-forecast, a re-stated month, an experiment read wrong — which is why quality spend is chronically under-funded.
What this approach costs
- A complete detector portfolio costs compute, engineering time and alert budget. Covering all thirteen everywhere is not affordable; covering them on the datasets whose wrongness would be expensive is.
- Strict enforcement at the boundary catches meaning failures early and rejects data — which turns a quiet corruption into a loud outage. That is usually the right trade and it is never a free one (Contract Enforcement).
- A taxonomy makes reasoning fast and makes it lossy. A real incident occasionally sits between two classes, and the checklist has to be a starting point rather than an authority.
Thirteen ways a data platform is wrong
Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.
Thirteen ways a data platform is wrong
Not thirteen tools and not thirteen incidents — thirteen distinct shapes of wrongness. Pick one and read the two columns that matter: what fires, and what stays green while it happens.
failure class
A capture gap — a connector down, a window missed, an API page never fetched. The records exist in the source and were never emitted, so nothing downstream can know they were expected.
| Check | What it asserts | Here |
|---|---|---|
| Completeness | Every order the source recorded for the period reached the serving table. | FAIL |
| Uniqueness | Each order id appears exactly once in the serving table. | pass green, and blind to this |
| Freshness | The newest complete record is recent enough for the decisions this table drives. | pass green, and blind to this |
| Validity | Every amount is non-null and parses as a number. | pass green, and blind to this |
| Distribution | The shape of the day resembles the days before it, per country and in total. | FAIL |
| Reconciliation | Revenue summed in the serving table equals revenue summed in the source for the same closed period. | FAIL |
what actually catches it
Reconciliation against the source for a closed period, and a source-side row count the pipeline does not compute itself.
what that still misses
A gap in a period that is not yet closed, and a gap that a duplicate elsewhere happens to offset in the total.
in the pipeline labThis class is a toggle in pipeline-lab; the fingerprint above is the one that lab produces, so the two can never drift apart.
SIMULATEDWhich checks fire is computed by running this fault through the pipeline model, not asserted by hand.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe thirteen classes are properties of moving data between systems and appear in every stack, including a single scheduled SQL script. What varies is which are frequent — a batch-only platform rarely sees out-of-order events until it adopts change capture.
- SIMPLIFIEDThirteen is a teaching cut, not a partition of reality: real incidents combine classes, and a schema change that produces a null cast that drops a sum is three rows of this table in one event. The cut is chosen so that each row has a distinct detector.
- SCALE-SPECIFICSkew and partial loads barely exist below the point where a job is distributed across workers or a dataset is split across partitions. On a single-node nightly script they are replaced by one class — the script did not finish — which is loud.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
Concurrencyordering-guarantees
Domains that do not exist yet
- — Distributed Systems owns why at-least-once is the default rather than a defect, why per-partition ordering is the strongest ordering a partitioned log can cheaply offer, and what a replay actually replays. Six of the thirteen classes are consequences of those results.
- — DevOps / Production Engineering owns the deployment side of the schema and logic classes: how a transformation change is reviewed, released and rolled back, and how to tell an incident caused by a deploy from one caused by data.