Lineage Debugging
Click a dashboard metric and walk it back — tile, metric definition, mart, model, staging, raw, change capture, production database — then turn around and ask what else this feeds.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Where did this number come from, and if the table behind it is wrong, what else is wrong right now?
Two people at once. The engineer in an incident, who needs the upstream path to find the fault; and the person deciding whether to change a column, who needs the downstream path to know who they are about to break (Impact Analysis).
One lineage edge: one dataset (or one column) reads from another. The graph is the transitive closure of those edges, and its usefulness depends entirely on whether the edges are generated by the tools that create them or written down by people who meant to.
Keep the lineage in a diagram in the team wiki, updated when someone remembers. It is accurate the week it is drawn, it reads well in onboarding, and it costs nothing to maintain because nobody maintains it.
The diagram shows six models and there are now nineteen. The five it is missing are the ones added under time pressure, which are exactly the ones involved in incidents (Data Platform Anti-Patterns).
- The diagram shows six models and there are now nineteen. The five it is missing are the ones added under time pressure, which are exactly the ones involved in incidents (Data Platform Anti-Patterns).
- A column is renamed and the impact analysis is done by grepping the repository. It finds every SQL model and none of the BI-tool queries, saved extracts, notebooks or scheduled exports (Breaking Schema Changes).
- The graph is at table level, so "which dashboards use
customer_email" is unanswerable, and a PII deletion request has to be treated as affecting everything (Deletion Requests). - During an incident the question is "what else does this feed", and the wiki answers "the dashboards". Forty consumers, no list, and the notification goes to a channel and hopes (Data Incidents).
- Two models compute revenue differently and both are correct by their own definitions. Without lineage nobody can see that the executive tile and the finance report do not share an upstream (The Metrics Layer).
What is actually happening
- Lineage is a directed acyclic graph over datasets, and the two questions it answers are the two directions of traversal. Upstream from a node is the debugging path; downstream is the blast radius. They are the same graph read in opposite directions and they are used by different people for different reasons (DAG (Directed Acyclic Graph)).
- The edges can be obtained three ways, in descending order of reliability: emitted by the transformation tool as it runs, parsed from the SQL that defines each model, or declared by a human. Emitted edges reflect what actually ran; parsed edges reflect what is written down; declared edges reflect what somebody believed in the past (Data Lineage).
- Column-level lineage is a different and much more useful object than table-level. "This dashboard depends on
fct_orders" is nearly always true and rarely actionable; "this tile depends onfct_orders.net_amount, which is derived fromraw.payments.amount_minor" is what makes an impact analysis a list instead of a shrug (Column-Level Lineage). - Every graph has a boundary, and the boundary is where lineage becomes fiction. A spreadsheet someone maintains, an extract emailed monthly, a notebook that reads the warehouse — all real consumers, none of them edges. Knowing where your graph stops is more useful than pretending it does not (Data Discovery).
- The metric definition is a hop that people skip. Between the serving model and the tile there is SQL — in a semantic layer, or pasted into the tool — that filters, joins and aggregates. It is a transformation with no tests, no version history and usually no lineage edge, and it is a genuine source of wrong numbers (Where Did This Number Come From?).
Walking a number back to a database row
The flagship interaction of this domain is small: click a number and see every hop that produced it, with what each hop holds and what each hop could have corrupted. Done well it replaces an afternoon of asking people who might know.
The chain below is the common shape, and the two hops at the top are the ones most often missing from a lineage implementation. The BI layer applies filters and joins of its own; the metric definition is SQL with no tests. Between them they produce a genuine share of wrong numbers, and neither is upstream of anything a data test can see (Where Did This Number Come From?).
The couldCorrupt column is what makes this a debugging tool rather than a diagram. During an incident you are not reading the path, you are reading that column and asking which entry matches the symptom — an understated total points at a different set of nodes than a doubled one (Data Incidents).
- Dashboard tile
holds One aggregated number plus whatever filters the BI tool applies at render time.
could corrupt A filter, a join or a cached extract added in the tool, invisible to every upstream test and to most lineage implementations.
↑ reads from - Metric definition
holds The SQL or semantic-layer expression that turns a model into the metric — the
where, thesum, the grain it aggregates to.could corrupt A definition that differs from the one another team uses, so two correct numbers disagree; or an aggregate at the wrong grain (The Metrics Layer).
↑ reads from - `revenue_daily` mart
holds One row per country-day with revenue pre-summed for query speed.
could corrupt A refresh that lags the model it derives from; a re-aggregation that double counts when someone joins it back to order level (Data Marts).
↑ reads from - `fct_orders`
holds One row per order, with measures and dimension keys at a declared grain.
could corrupt A fan-out join against a dimension with duplicate keys; a filter that silently drops a status; logic that stopped netting refunds (Grain: What Does One Row Represent?).
↑ reads from - `stg_orders`
holds One row per order, reconstructed as the latest change per order id.
could corrupt Choosing "latest" by arrival rather than commit order; failing to apply deletes; a cast that nulled a measure (CDC Ordering and Transaction Boundaries).
↑ reads from - Raw change files
holds Every delivered change record, exactly as received, immutable.
could corrupt Duplicates from redelivery, and a gap where the connector was down long enough for retention to expire (The Raw Landing Zone).
↑ reads from - Change capture connector
holds A position in the source database's write-ahead log.
could corrupt Falling behind retention; restarting from a snapshot and re-emitting history; missing DDL changes entirely (CDC Failure Modes and the Retention Deadline).
↑ reads from - Production `orders` table
holds The authoritative current state of every order.
could corrupt Nothing, from this domain's point of view. If the number is wrong here it is an application incident, and finding that out is a successful walk (Source of Truth).
Eight nodes, and the fault can be at any of them. What makes the walk fast is that each node's checks for the affected period are attached to it, so the path is annotated with state rather than being a picture.
The same graph, read the other way
Turn around at any node and the graph answers a different question for a different person: what depends on this. Upstream is for the incident; downstream is for the change, the rebuild order and the notification list (Impact Analysis).
The downstream direction is used badly more often than the upstream one, because at table level it over-answers. "Everything that reads fct_orders" is most of the warehouse, and an impact analysis that returns most of the warehouse is one people learn to skip. Column-level edges are what make the answer a list of nine rather than a shrug.
The fan-out in the diagram is the honest shape of a real platform: one raw source feeds several models, one model feeds several marts, and one mart feeds many tiles. It also shows the boundary — the extract and the notebook that read the warehouse through nothing that emits an edge, and are therefore consumers your graph will never mention (Data Discovery).
Impact analysis, and the consumer that breaks silently
The highest-value use of the downstream direction is before a change, not during an incident. A producer is about to rename a column; the question is who breaks, and — much more important — who breaks *silently*.
The loud breaks take care of themselves: a query referencing a column that no longer exists fails, someone is paged, the change is reverted. The dangerous rows in the table below are the ones marked silent, where the consumer keeps running and starts producing a different number. Those are the ones that reach a quarterly report (Breaking Schema Changes).
Lineage produces the list; it does not produce the silent-versus-loud judgement, which requires knowing how each consumer uses the column. That is why an impact analysis ends in a conversation and why column-level edges plus documented ownership are worth more together than either is alone (Data Ownership).
- order_id
- customer_id
- order_ts
- status
- amount_cents
- currency
- order_id
- customer_id
- order_ts
- status
- amount_minor
- currency
change A column is renamed at the source. The type is unchanged, the meaning is unchanged, and the migration is correct by the producing team's standards.
| Consumer | Effect | How it shows up |
|---|---|---|
| `stg_orders` model, referencing `amount_cents` explicitly | The model fails to compile on the next run. Loud, immediate, attributable, and the cheapest possible outcome. | Loudly — it raises |
| `stg_orders` model, written with `select *` | The model succeeds and the column silently changes name downstream, so every model that referenced amount_cents breaks one layer later with no obvious cause. | Silently — no error, wrong result |
| A raw ingestion layer that lands unknown fields | The new column is landed and the old one stops appearing. Row counts are unaffected; the measure is simply absent from that day forward (The Raw Landing Zone). | Silently — no error, wrong result |
| BI tile with the column pinned in a saved query | The tile errors or renders empty, depending on the tool. Sometimes loud, sometimes an empty chart nobody looks at until month end. | Silently — no error, wrong result |
| Monthly finance extract maintained outside the platform | Breaks or silently produces nulls, and nobody on the data team finds out because the consumer is outside the lineage graph entirely. | Silently — no error, wrong result |
| Freshness and validity checks defined on `amount_cents` | The checks fail to run or evaluate an absent column. The monitoring goes quiet at exactly the moment it was needed (Data Observability). | Silently — no error, wrong result |
How to build it
Most important first.
- Generate lineage from execution or from parsing, never from memory. A graph that requires human maintenance is out of date within a month and dangerous within two, because people will trust it (Metadata: Technical, Operational and Business).
- Push for column level where it is affordable. The step from table-level to column-level lineage is what turns "everything might be affected" into a list of nine consumers (Impact Analysis).
- Extend the graph past the warehouse into the BI layer, so tiles and saved queries are nodes. The last hop is where a surprising share of wrong numbers are created and it is the hop most often left out (The Data Catalog).
- Attach checks and freshness to the nodes, so the walk shows not only the path but the state of every node on it for the affected period. That is the difference between a diagram and a debugger.
- Record the boundary explicitly: which consumers are known to exist outside the graph. An honest edge list plus a named blind spot beats a complete-looking graph that quietly omits the finance spreadsheet.
- Use the downstream direction *before* changes, not only during incidents. Impact analysis at review time is the cheapest moment to discover that a rename breaks a report nobody remembered (Data Contracts).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Emitted lineage guarantees the edges that ran, for the runs that were observed. It does not include a consumer that reads the table outside your tooling, and it cannot.
- Parsed lineage guarantees the edges the SQL declares. Dynamic SQL,
select *and string-built queries all degrade it, andselect *degrades it precisely by erasing the column-level information that made it valuable. - A complete-looking graph guarantees nothing about completeness. Absence of an edge is evidence of absence only within the tools that emit edges (Data Discovery).
- Lineage guarantees nothing about correctness. It says one dataset feeds another, not that the transformation between them is right — which is why it detects nothing on its own and makes every other signal actionable (Data Observability).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check: assert that every dataset in the catalog appears in the lineage graph, and that every BI tile resolves to at least one dataset. Orphans in either direction are the graph telling you where it is blind.
- It misses consumers outside the tooling entirely — the extract, the notebook, the spreadsheet — which are exactly the consumers who will not be notified during an incident.
- It also misses edges that exist but are wrong: a parsed edge from a model that was deleted, or a stale edge kept alive by a definition that no longer runs (Dataset Documentation).
- The graph must be as fresh as the pipelines it describes. A lineage graph refreshed weekly will not contain the model deployed on Tuesday, which is the model most likely to be involved in Wednesday's incident.
- Node state is a separate freshness question from graph structure. During an incident you need each node's checks *for the affected period*, which may be days old, not its state now (Debugging a Data Incident).
- Historical lineage matters for reproducing an old number. "Which model produced this figure in March" requires the graph as it was in March, and almost no implementation retains that (Keeping Raw History: The Recovery Position and the Liability).
- A column rename is the canonical use of the downstream direction, and the reason it must be column level: table-level lineage reports every consumer of the table, which is unhelpful enough that people stop asking (Schema Evolution).
- A type change is a harder impact analysis than a rename, because consumers that break silently — a cast to null, a numeric comparison that becomes a string comparison — are not visible in any lineage edge. Lineage gives you the list; the silent-versus-loud judgement is still human (Nullability & Defaults).
- Semantic change is invisible to lineage in principle. The edges are unchanged, the types are unchanged, and every consumer of the column is now wrong. Lineage tells you who to talk to, which is the entire contribution it can make (Semantic Changes).
- After a fault is fixed at one node, the downstream direction gives the rebuild order: everything reachable from the repaired node must be recomputed, in topological order (Topological Execution).
- That same traversal gives the notification list, which is the half people skip. Correcting a model without telling the forty consumers who read yesterday's version means forty people hold a number they believe is current (Data Incidents).
- For deletion requests the downstream direction is the compliance requirement rather than a convenience: a row removed from a source persists in every derived dataset until each one is rebuilt (Deletion Requests).
What can go wrong
- A hand-maintained graph that is confidently wrong, which is worse than no graph because it is trusted.
- Table-level lineage used for column-level questions, producing impact analyses so broad that people stop running them.
- A graph that stops at the warehouse boundary, so the BI layer — where a real share of wrong numbers originate — is invisible.
select *in models, which erases column-level lineage wherever it appears and does so silently (CSV, JSON and Their Limits).- Consumers outside the tooling who are structurally unnotifiable, and a graph that does not admit they exist.
- A lineage product adopted without ownership, so the graph is complete, correct and consulted by nobody (Data Ownership).
- "We have lineage, so we have impact analysis." Only if it is column level and only if it extends to consumers. Table-level lineage answers "the whole warehouse may be affected", which nobody can act on (Impact Analysis).
- "Lineage is documentation." It is a debugging tool and a change-safety tool. Documentation is read at onboarding; lineage is queried under pressure, which is a much higher bar for accuracy (Data Lineage).
- "If it is not in the graph, nothing reads it." The graph knows about the tools that emit edges. The monthly extract, the notebook and the spreadsheet are real consumers and structurally invisible.
- "Lineage tells us what broke." It tells you what connects to what. Every node still needs its own checks for the affected period before the graph becomes a diagnosis (Data Observability).
- Column-level lineage is the mechanism that makes classification enforceable: a column tagged as personal data propagates its tag along every edge, and any derived dataset that is not similarly protected becomes visible as a gap (Data Classification).
- It is also the only practical way to answer a deletion request honestly, because the answer is "every dataset reachable downstream of this column" and that set is not knowable from memory (Deletion Requests).
- The graph itself is sensitive. It is a precise map of where personal data lives and which extracts carry it off the platform, and it deserves the access controls of the data it describes (Data Access Control).
Operating it
- The upstream path from any tile to a source table, with each node's freshness and check results for the period in question attached (Debugging a Data Incident).
- The downstream closure of any dataset or column, which is simultaneously the blast radius and the notification list.
- Coverage: the share of datasets and tiles that appear in the graph at all, trended. It is the only measure of how much the graph can be trusted.
- Edge age — when each edge was last confirmed by an actual run. A stale edge is a claim about a pipeline that may not exist any more.
- At ten times the datasets, the graph stops being drawable and becomes a query target. Nobody looks at a picture of nineteen hundred nodes; they ask it questions (Data Discovery).
- At a hundred times, the downstream closure of a widely-used dimension is most of the platform, and the useful answer needs ranking — by tier, by consumer, by recency of use — rather than completeness (Impact Analysis).
- Column-level edges grow with columns rather than with rows, so the graph is stable under data growth and jumps whenever a wide model is added. That makes its cost predictable in a way little else here is.
- Graph storage is negligible; the cost is in producing edges. Parsing every model on every change is cheap, and column-level resolution through joins and CTEs is meaningfully harder and is where implementations differ most (The Planner: Enumerating Ways to Answer).
- Extending into the BI layer costs an integration per tool and is usually the highest-value increment available, because it covers the hop that creates wrong numbers with no tests behind it.
- Retaining historical versions of the graph multiplies storage by the number of retained versions, which is small in absolute terms and is the only way to reproduce how an old number was made (Storage Lifecycle).
- Column-level lineage answers the questions that matter and is substantially harder to produce accurately, particularly through joins, unions and dynamic SQL. Table-level is cheap and answers a coarser question, and pretending it is the same thing is how impact analysis loses its audience.
- Emitted lineage is accurate about what ran and blind to what has not run recently. Parsed lineage covers everything defined and includes paths that never execute. Running both and reconciling them is the honest option and it is two systems.
- Extending the graph into every consumer surface is where completeness gets expensive. At some point the right answer is a named boundary and a documented blind spot rather than another integration.
Lineage debugger
Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.
One number, with the grain now invisible.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe two traversal directions and their two audiences are properties of a dataflow graph and hold everywhere. What varies enormously is edge quality: a SQL-first platform can parse most of its graph, while one built from imperative jobs can often only observe reads and writes.
- TOOL-SPECIFICTransformation frameworks that model datasets as first-class objects emit lineage as a by-product of running, while orchestrators that model tasks can only report which process ran — so the same platform yields a column-level graph in one arrangement and nothing at all in the other.
- ORG-SPECIFICThe graph stops at the boundary of the tools a team controls. Where analysts extract to spreadsheets or notebooks routinely, the most complete technical lineage still misses the consumers most likely to be surprised, and the remedy is organisational rather than technical.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why the causal graph of a distributed computation is harder to reconstruct than a dataflow graph, and what it costs to record one faithfully.
- — DevOps / Production Engineering owns change management: a lineage-driven impact analysis is only preventive if it runs as part of code review rather than after deployment.