Data Lineage
orders DB to stg_orders to fct_orders to revenue_daily to the executive dashboard — and why that graph is a debugging tool rather than documentation.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
A number on a dashboard looks wrong. What produced it, what produced that, and where does the walk stop?
An engineer in the first ten minutes of a data incident, who has one question — what feeds this — and will ask it repeatedly, upstream, until an answer changes. Secondarily a producer about to change something, who needs the same graph read in the opposite direction (Impact Analysis).
One lineage edge is one dependency between two datasets, produced by one job. The job part is what makes it useful: an edge without a producer tells you that A relates to B, while an edge with a producer tells you which code to open. Coarser grains — team-level or system-level lineage — look tidier in a diagram and cannot answer an incident question (Column-Level Lineage).
Draw the pipeline in a diagramming tool during the design review, put it in the team wiki, and link to it from the runbook. It is accurate on the day it is drawn and it is genuinely useful for onboarding.
The diagram was drawn for the design and the implementation diverged in week three. Nothing forces the two back together, and the difference is invisible until an incident, when the diagram sends an engineer to a model that no longer feeds the affected table.
- The diagram was drawn for the design and the implementation diverged in week three. Nothing forces the two back together, and the difference is invisible until an incident, when the diagram sends an engineer to a model that no longer feeds the affected table.
- A new model is added between
stg_ordersandfct_orders. Nobody updates the diagram, because updating the diagram is not part of merging the model (The Transformation DAG). - During the incident the actual question is "which of the eleven things downstream of
fct_ordersare affected", and a hand-drawn diagram of the happy path does not contain the downstream fan-out at all (Impact Analysis). - The diagram shows systems — "Postgres → Lake → Warehouse → BI" — at a grain far too coarse to act on. Every incident is inside one of those boxes (Where Did This Number Come From?).
- A dashboard is built directly on a staging model by an analyst who had permission and a deadline. That edge exists in production and in nobody's diagram, and it is the edge that breaks when staging is refactored.
What is actually happening
- Lineage is a directed graph whose nodes are datasets and whose edges are jobs. It is not a picture of your architecture; it is the transitive closure of "reads from" over everything that actually ran. That distinction is the whole lesson, because a picture is drawn once and a closure is recomputed continuously (The Transformation DAG).
- Edges are produced three ways, and they have very different coverage. Parsed: read the SQL a job executed and resolve which tables it selected from and which it wrote. Declared: the transformation framework already knows, because the model references its parents explicitly. Observed: read the engine's own query history and infer edges from what actually ran (Follow the Query).
- The direction of reading matters more than the graph itself. Upstream answers "what could have caused this", and is walked one hop at a time during an incident with a single question at each node: is the affected period complete and correct *here*? The first "no" going upstream is where the incident lives (Debugging a Data Incident).
- Downstream answers a different question — "what does this break" — and is the direction that decides blast radius before a change instead of after it. Same graph, reversed edges, completely different use (Impact Analysis).
- This is why lineage is a debugging tool and not documentation. Documentation is read while you are calm, is allowed to be approximate, and describes intent. Lineage is read at 03:00 by someone who has been told a number is wrong, must be exact about what actually ran, and is useless if it describes what was supposed to happen (Lineage Debugging).
- The corollary: lineage must be generated, and generated by the thing that runs the work. Any lineage that requires a human step to stay current is a diagram with extra steps, and will be wrong in precisely the situations that motivated building it.
The chain, and what each hop can corrupt
Here is the canonical chain the rest of this module refers to. Five hops from an operational database to a number an executive reads, each hop a separate system with its own failure behaviour.
The value of writing it as a table rather than as a picture is the third column. A diagram tells you the order; what an investigator needs is the list of things each node is capable of getting wrong, because that is what turns "walk upstream" into "check this specific property at this node".
Walk it from the bottom of the incident to the top. At each node ask exactly one question — is the affected period complete and correct here? — and stop at the first node where the answer changes from no to yes. The node below that boundary is where the incident lives (Debugging a Data Incident).
- `orders` (operational database)
holds The authoritative current state of every order, as the application wrote it.
could corrupt Nothing from this domain's point of view. If the number is wrong here it is an application incident, and the walk has terminated successfully (Source of Truth).
↑ reads from - Ingestion / CDC
holds A position in the source log, and the change records emitted from it.
could corrupt Gaps where the connector was down past log retention; duplicates after a restart; a snapshot that re-emitted history; DDL changes it did not carry (CDC Failure Modes and the Retention Deadline).
↑ reads from - `stg_orders`
holds One row per order, reconstructed from change records and lightly cleaned.
could corrupt Choosing "latest" by arrival rather than commit order; dropping deletes; a cast that nulls rather than errors (CDC Ordering and Transaction Boundaries).
↑ reads from - `fct_orders`
holds One row per order at a declared grain, with measures and dimension keys.
could corrupt A fan-out join against a dimension with duplicate keys; a status filter that silently excludes a category; an order-line join that multiplies every order-level measure (Grain: What Does One Row Represent?).
↑ reads from - `revenue_daily`
holds One row per country-day with revenue pre-aggregated.
could corrupt Aggregating at the wrong grain; a refresh that lags the fact table it derives from; a partial publish that exposes half a day as a complete one (Atomic Publish).
↑ reads from - Executive dashboard
holds One number, plus filters and joins defined inside the BI tool.
could corrupt A filter or blend added in the BI layer that no model, test or lineage edge can see — the most common last hop and the least instrumented (Two Dashboards, Two Numbers).
The couldCorrupt column is the checklist. It is also why lineage at system granularity — "Postgres → Lake → Warehouse → BI" — is useless during an incident: every entry in that column happens *inside* one of those four boxes.
A debugging tool, not documentation
These two artefacts look similar and are used at opposite ends of the emotional spectrum. Documentation is read deliberately, by someone with time, who will tolerate approximation because they are building context. Lineage is read reactively, by someone who has just been forwarded a screenshot of a wrong number, who needs the graph to be exact about what ran rather than eloquent about what was intended.
That difference dictates every design decision. It is why lineage must be generated rather than written; why the unit is the job rather than the team; why the interface is a one-hop-at-a-time walk rather than an overview; and why an edge that describes the design rather than the execution is not a lesser version of lineage but the opposite of it.
It also explains the most common disappointment with lineage projects. A team builds a beautiful full-graph visualisation, demonstrates it, and then during the first real incident nobody opens it — because the incident question is "what feeds this one node" and the tool answers "here is everything".
A diagram maintained by hand, updated when someone remembers, showing systems and the intended flow between them. Reviewed at design time, linked from the wiki, rendered at whole-platform granularity.
Edges emitted by the jobs themselves at execution time, keyed by dataset and job, stored with run identifiers and timestamps, queried as "one hop upstream from this node" and "everything downstream of this node".
The question lineage is asked is always about a specific dataset in a specific period, and the answer must reflect the code that actually executed. A hand-maintained picture diverges from execution within weeks, and its divergence is undetectable — which means during an incident it does not merely fail to help, it actively misdirects the walk.
1{2 "eventType": "COMPLETE",3 "eventTime": "2026-03-14T02:17:44Z",4 "run": { "runId": "8f3c1a4e-..." },5 "job": { "namespace": "warehouse", "name": "build_fct_orders" },6 "inputs": [7 { "namespace": "warehouse", "name": "analytics.stg_orders" },8 { "namespace": "warehouse", "name": "analytics.dim_customer" }9 ],10 "outputs": [11 {12 "namespace": "warehouse",13 "name": "analytics.fct_orders",14 "facets": {15 "rowCount": 1842097,16 "partition": "order_date=2026-03-13",17 "codeVersion": "9c21ef0"18 }19 }20 ]21}Three things make this an incident tool rather than a diagram. The runId lets you find the logs. The codeVersion lets you ask what changed. The partition on the output means a later question about one specific bad day resolves to one specific run rather than to "the job that builds this table".
OpenLineage is an open specification for events of roughly this shape, and several orchestrators and transformation tools can emit it. Which tools emit it natively, and which facets they populate, changes release to release — verify current documentation rather than planning around a facet you have not seen produced.
Three ways to produce edges, and what each costs
Nobody gets complete lineage from one source. Declared lineage is exact and covers only the framework; parsed lineage covers whatever the parser can understand; observed lineage from query history covers everything that ran and attributes some of it wrongly. Real platforms run at least two and reconcile.
The reconciliation is itself the most valuable artefact. Edges that appear in the query log and not in the declared graph are the paths built outside the sanctioned framework — which are, reliably, the paths that break during a refactor and the ones nobody thought to warn (Data Platform Anti-Patterns).
The cost shape below ranks the effort. Note that the largest bar is not any of the generation methods: it is covering the BI layer, because that boundary usually requires a separate integration per tool and it is the hop where the number people complain about is actually computed.
A separate integration per tool, each with its own model of what a "dataset" is — and the hop where the reported number is often actually computed.
Needs a real parser per dialect and still fails on dynamic SQL, stored procedures and user-defined functions; grows with query volume rather than data volume.
Turns one current-state graph into an append-only history; the storage is modest and the query complexity is not.
Often close to free because the framework already resolves dependencies to build its execution order; covers only models inside the framework.
Cheap, and coarse: it knows task dependencies, which are not always dataset dependencies — two tasks ordered for convenience produce a false edge.
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative effort in a typical warehouse-centric platform, shown to establish an ordering rather than as a measurement. The teaching is the ordering: the cheapest source is the one that covers least, and the boundary everyone under-invests in is the last hop into the tool where the complaint originated.
What kind of incident are we currently unable to investigate?
when Most transformation already lives in one framework and incidents are about models within it.
cost Lowest effort, and it silently excludes everything outside the framework — which will include the exports and the notebooks (dbt Concepts).
when Substantial work happens outside the framework, or you need to measure how much does.
cost Parsing effort, dialect handling, and noise from ad-hoc queries that create edges nobody wants shown (Follow the Query).
when Incidents keep terminating at "the dashboard does something we cannot see".
cost One integration per BI tool, ongoing maintenance, and the highest value per unit of effort in most platforms (Two Dashboards, Two Numbers).
when Table-level lineage keeps answering "this model" when the question was "this column".
cost Substantially harder to produce and substantially more useful during an incident. A separate decision, not an increment (Column-Level Lineage).
when Under a few dozen models, one team, and everyone can name the dependencies from memory.
cost None, and the memory expires quietly. The moment to build is when the first person who cannot name them joins.
How to build it
Most important first.
- Emit lineage from the execution path — the orchestrator or transformation framework that already knows the dependency, at the moment it runs. Generated edges are the only ones that survive a refactor nobody told you about (dbt Concepts).
- Include the job and the run on every edge, not just the two datasets. During an incident the question after "what feeds this" is always "which run wrote the affected partition, and with what code version" (Orchestration).
- Cover the last hop into the BI layer. Lineage that stops at the warehouse table stops one hop short of where the number that triggered the incident actually lives, and the BI layer is where a surprising share of transformation logic hides (Two Dashboards, Two Numbers).
- Backfill lineage from the query history for everything outside the framework — ad-hoc scripts, notebooks, exports. Observed edges are noisier than declared ones and they are the only way to see the paths nobody declared (Data Discovery).
- Store lineage with time. "What fed this table" has a different answer for last month's partition than for today's, and an incident about a historical period needs the historical graph (Reprocessing vs Retrying).
- Make the upstream walk the primary interface, because that is the direction of the question people arrive with. A graph visualisation that shows everything at once is impressive and is not how anyone debugs (Lineage Debugging).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Generated lineage guarantees that the edges it shows really ran, for the runs it observed. It never guarantees the absence of an edge: a path that was never instrumented produces no edge and looks identical to no dependency.
- Parsed lineage guarantees only what the parser understood. Dynamic SQL, string-built queries, stored procedures and user-defined functions routinely produce a job with inputs the parser could not resolve, and most parsers report that as fewer edges rather than as an error (Column-Level Lineage).
- Observed lineage from query history guarantees coverage of what ran within the log's retention window. Outside that window the graph is silent, which matters for anything quarterly.
- Lineage guarantees nothing about correctness. An edge says data flowed; it says nothing about whether the transformation on that edge preserved meaning, grain or completeness (Grain: What Does One Row Represent?).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Check for orphan nodes: datasets in the catalog with no inbound lineage edge. Each one is either a source, a manually loaded table, or a gap in instrumentation, and the first two are small enough to enumerate — so anything left is a coverage hole (The Data Catalog).
- Check the graph against the query log. Any table pair that appears in a real query and not in the lineage graph is an edge your generator missed, and this reconciliation is the only way to measure lineage recall rather than assume it.
- Both miss the edges that leave the platform entirely — a scheduled export to a spreadsheet, a report emailed monthly, a reverse-ETL sync to a CRM. Those dependencies are real, break loudly, and are invisible to every generator (Data Platform Anti-Patterns).
- Lineage should be as fresh as the last run of each job, because that is when the dependency was last true. A graph refreshed nightly is a graph that cannot describe a pipeline changed this morning — which is the pipeline most likely to be in an incident.
- The graph and the data have different freshness and both matter. An edge that is current while the dataset it points at has not been written for three days is exactly the situation an incident walk is trying to surface (Freshness Monitoring).
- Historical lineage has no freshness requirement and a retention one instead. The question "what fed this table in March" is asked during backfills and audits, and is unanswerable if the graph is stored as current state only (Backfills).
- A refactor that splits or merges models rewrites large parts of the graph in one commit. Generated lineage absorbs that automatically; any manually maintained representation of it does not, and the divergence begins immediately (Model Layering).
- A rename creates a new node and orphans the old one unless nodes are keyed by something stable. The visible symptom is a dataset that appears to have no history and no downstream, which reads as "safe to drop" (Schema Evolution).
- When the graph itself changes shape, historical incidents become harder to reason about. Storing lineage versioned by time is what lets an investigator ask what the pipeline looked like when the bad partition was written.
- Lineage is derived data and is rebuildable from run history and query logs, bounded by their retention. This makes the lineage store safe to drop and re-derive, which is worth knowing before an upgrade.
- During a backfill, lineage is what tells you the set of downstream datasets that must also be recomputed and in which order. Recomputing a fact table without its dependents leaves the platform internally inconsistent in a way that is very hard to detect (Planning a Backfill).
- Topological order comes straight from the graph, which is why the DAG algorithms matter here rather than being trivia: the recompute order for a backfill is a topological sort of the affected subgraph (Topological Execution).
What can go wrong
- Lineage stops at the warehouse boundary, so the BI layer where the reported number is actually computed is outside the graph and outside the investigation (Two Dashboards, Two Numbers).
- A parser silently fails on a subset of jobs and reports fewer edges rather than an error, so coverage degrades invisibly and confidence in the graph stays high.
- The graph is complete and unusable: thousands of nodes rendered at once, with no way to walk one hop at a time from the node the incident started at.
- Lineage is generated only for successful runs, so a job that failed halfway — having written some outputs — leaves no edges for what it did write.
- The lineage service becomes a dependency of the deploy pipeline and its outage blocks releases, which is how an observability tool turns into an availability risk.
- "Lineage is documentation." It is read during incidents, by people under pressure, one hop at a time, and it must describe what ran rather than what was designed. Documentation may be approximate; this may not (Dataset Documentation).
- "We have a lineage diagram." A diagram is a snapshot of an intention. Lineage is a generated closure over what actually executed, and the two disagree within weeks (The Transformation DAG).
- "The graph is complete." It covers what was instrumented. Exports, notebooks, reverse-ETL syncs and hand-run scripts are real edges with no representation, and treating absence of an edge as absence of a dependency is how a "safe" deletion breaks a finance report (Impact Analysis).
- "Lineage tells us the data is correct." It tells you what flowed where. A fan-out join, a wrong filter and a correct transformation all produce the same edge (Grain: What Does One Row Represent?).
Operating it
- Lineage coverage: share of catalogued datasets with at least one inbound edge, trended. A falling line means new pipelines are being built outside the instrumented path (The Data Catalog).
- Edge freshness distribution — how old the newest edge is per dataset. Datasets whose most recent edge is weeks old are either abandoned or produced by something you are not watching.
- Depth from source to each serving dataset. Growing depth is a real signal about platform complexity and about how long an incident walk will take (Debugging a Data Incident).
- Count of edges discovered from query logs that the generator did not know about. This is the recall gap, stated as a number rather than as a feeling.
- At ten times the model count the graph stops being viewable as a whole. The interface must become "walk from this node", and any product that only offers a whole-graph rendering has quietly stopped being usable (Lineage Debugging).
- At a hundred times, the interesting queries are graph queries — ancestors, descendants, shortest path between a dashboard and a source — and the storage choice starts to matter. This is where the graph traversal being a real algorithm rather than a diagram becomes concrete (Directed Graph).
- Consumer growth increases the downstream fan-out far faster than upstream depth. Upstream walks stay short; impact analysis gets harder every quarter (Impact Analysis).
- Emission cost is negligible — a lineage event is small and is written once per run. The cost is in storage and query of the accumulated graph, which grows with runs times edges rather than with data volume (What Actually Drives Data Platform Cost).
- Parsing query history to infer edges is the expensive path: it means reading and analysing every query the platform ran, and the cost scales with query volume, which in a busy warehouse is large (Scan Cost).
- The cost that actually justifies the work is the one avoided: incident time spent working out what feeds what. That cost is paid in the worst possible currency, which is engineer attention during an outage.
- Generated lineage is accurate and covers only instrumented paths. Choosing it means accepting that everything outside the framework is invisible until you add a second, noisier source — and that the graph will therefore always be slightly optimistic about completeness.
- Storing lineage over time makes historical investigation possible and multiplies the storage and the query complexity. Most platforms store current state only and rediscover the need for history during their first serious backfill (Backfills).
- Table-level lineage is cheap to produce and answers "which model" but not "which column", which is the question an incident usually needs. Column-level costs far more to produce and is far more useful, and that is a real trade rather than an upgrade (Column-Level Lineage).
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe upstream walk as an incident method works with a hand-drawn graph and no tooling at all. What a lineage system changes is speed, and the ability to answer the reverse question about blast radius, which is impractical by hand above a few dozen datasets.
- TOOL-SPECIFICCoverage depends entirely on what produces the edges: a framework that models dependencies explicitly emits complete lineage for its own models and nothing for anything else, while query-log parsing sees everything that ran and mis-attributes dynamic SQL. Most platforms need both and should measure the gap between them.
- SIMPLIFIEDThe five-node chain in this lesson is a teaching shape. Real graphs fan out — one staging model feeding twenty marts — and fan in, and the interesting incidents are almost always at a node with several parents where only one of them is wrong.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why a per-hop record of what happened is the only workable substitute for a global view of a system whose components never agree on a single instant. Lineage is that idea applied to datasets rather than to requests.
- — DevOps / Production Engineering owns the change side: relating a bad partition to the deploy that caused it needs a deploy record joined to the run record, and that join is the whole trick.