Data Pipeline Visualizer
Source → change capture → broker → raw → transform → warehouse → dashboard. Turn on a fault and read two things: the number the dashboard reports, and the number that is actually true. The gap between them is the whole subject of this domain.
Every value below comes from a model in this repository — src/de/sim/pipeline.ts — run over a deterministic set of generated orders. Nothing was measured on a real platform, and a real platform would not give you these figures anyway. There are no prices, because a price would belong to somebody else's contract. What transfers is the shape: which stage loses the rows, and which check would have noticed.
The invariant that makes this a model rather than a prop: with no faults on, reported equals truth exactly. Every divergence you see afterwards is attributable to something you turned on. That is not how production feels, and it is exactly why this is worth doing first — you get to learn the signatures while you still know the answer.
The second thing to watch is the check row. No two faults here fail the same set of checks. That is not a convenience of the model; it is the reason diagnosis is possible at all, and it is what a real platform is trying to buy when it invests in tests rather than in alerting on task status.
The lab
Toggle faults on the left, mitigations underneath, and read the dashboard number against the truth.
| Stage | One row is | Rows in | Rows out | Δ |
|---|---|---|---|---|
Source database | One order, in its current state. | 4,000 | 4,000 | — |
Change capture | One committed change to one order. | 4,000 | 4,000 | — |
Event log | One delivered change record — possibly delivered more than once. | 4,000 | 4,000 | — |
Raw landing | One line in an immutable file, exactly as received. | 4,000 | 4,000 | — |
Transformation | One order, deduplicated and windowed. | 4,000 | 4,000 | — |
Serving table | One order, with measures and dimension keys. | 4,000 | 4,000 | — |
Dashboard | One number, with the grain now invisible. | 4,000 | 1 | aggregated |
Row counts are simulatedsim. The last row is where the grain disappears: one number, with nothing on the screen recording what one row of the source meant.
| Check | Result | What the model found | Still misses |
|---|---|---|---|
Completeness Every order the source recorded for the period reached the serving table. | pass | Every order in the source for this period is present. | Duplicates that coincidentally offset losses, and any period that is not yet closed. |
Uniqueness Each order id appears exactly once in the serving table. | pass | Every order id appears exactly once. | A genuine duplicate that arrived under a new key — a producer retry with a fresh event id looks like a second order. |
Freshness The newest complete record is recent enough for the decisions this table drives. | pass | Newest complete record is 12 simulated minutes old. | Data that is perfectly fresh and completely wrong. It also fires falsely on a period where the source genuinely produced nothing. |
Validity Every amount is non-null and parses as a number. | pass | Every amount is non-null and numeric. | A value that is well-typed and wrong — a price in the wrong currency passes every type check there is. |
Distribution The shape of the day resembles the days before it, per country and in total. | pass | Largest per-country share drift 0.8pp; total volume drift 0.0%. | Slow drift, and any error that preserves the shape while changing every value inside it. |
Reconciliation Revenue summed in the serving table equals revenue summed in the source for the same closed period. | FAIL | Serving table reports 1,010,654.50 against a source total of 934,498.90. | Anything wrong identically at both ends — a bug in logic shared by the extract and the model reconciles perfectly. |
- 1Refunded orders were counted at their full value. Every row is present, unique, fresh and well-typed — and revenue is overstated.
The revenue model stops subtracting refunds.
The code does exactly what it was told, and what it was told is wrong. Every row is present, unique, fresh, well-typed and normally distributed — and the number is too high.
The eight faults, and where each one is taught properly
Open one to see what is really happening underneath the symptom, and which signal separates it from the others.
Where to go next
Each fault above has a page of its own where it is the whole subject.