Dataset Documentation
Documentation generated from the transformation graph stays true. Documentation written separately does not — and the distinction decides what is worth writing down at all.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Which parts of a dataset's documentation will still be true in a year, and which will quietly become lies?
Someone who has already found the dataset and now has to decide whether it answers their question — what one row is, what is excluded, which definition of the metric applies, what is known to be wrong with it. Discovery got them here; this is what stops them using it incorrectly (Data Discovery).
A documentation unit is one claim about one object, at the same granularity as the object: a table-level claim about grain and scope, a column-level claim about meaning and units. Table-level prose describing column semantics is the shape that rots fastest, because nothing about changing a column touches it (Column-Level Lineage).
Write a good document for each important dataset — a wiki page or a README with grain, definitions, caveats and examples. Done properly it is the most useful artefact on the platform, and the first version genuinely is.
The model is changed and the page is not, because nothing about merging a change to the SQL requires touching a page in another system. The divergence starts on day one and is undetectable (The Transformation DAG).
- The model is changed and the page is not, because nothing about merging a change to the SQL requires touching a page in another system. The divergence starts on day one and is undetectable (The Transformation DAG).
- The page describes a column that was dropped two quarters ago, and a reader who cannot find it in the table assumes they are looking at the wrong table.
- A column added last month is absent from the page, so the documentation is simultaneously wrong about what exists and silent about what was added (Schema Evolution).
- The caveat that mattered — refunds arrive weeks after the order and land in the original date partition — was in a paragraph the reader skimmed, because it was surrounded by column descriptions that duplicate the schema (Late-Arriving Data).
- Two pages exist for the same model, written by different teams, and they disagree about whether revenue is gross or net. Both look official (Semantic Changes).
What is actually happening
- Documentation splits by whether it can be derived from something that changes when the data changes. Schema, column list, types, lineage, freshness, test coverage and the transformation expression itself are all derivable. Grain, exclusions, business definitions, known gaps and intended use are not (Metadata: Technical, Operational and Business).
- Derived documentation is true by construction and stays true without anyone maintaining it, because it is regenerated from the same source that produces the data. A column dropped from the model disappears from the documentation in the same commit (dbt Concepts).
- Written documentation has no such mechanism, so the question is not how to keep it fresh but how to make it small and how to put it where changing the logic makes the staleness visible. Co-location is the entire trick: a description in the same file as the SQL appears in the diff a reviewer reads (Data Contracts).
- The highest-value written claim is almost always the grain — what one row represents. It is the claim that decides whether an aggregate is correct, it cannot be derived, and it is the one most often missing (Grain: What Does One Row Represent?).
- The second is what is excluded. Filters are visible in the SQL and invisible in the output: a model that excludes test accounts, cancelled orders and internal transfers looks identical to one that does not, and the difference is the whole reason two teams get different revenue numbers (Two Dashboards, Two Numbers).
- Documentation that is also a test is the strongest form available. A stated set of accepted values that is asserted on every run is a claim that cannot drift, because the run fails when it stops being true. This is the only mechanism that gives written prose the durability of derived content (Data Tests).
Generated stays true; written drifts
The reliability of a documentation claim is decided almost entirely by whether changing the data requires changing the claim. That is a mechanical property, not a cultural one, and it means the useful question about any sentence in a dataset's documentation is: what would have to happen for this to become false without anyone noticing?
For a generated column list, the answer is nothing — the list is regenerated from the model on every build, so it cannot diverge. For a paragraph on a wiki, the answer is a one-line change to a WHERE clause made by someone who has never seen the page.
This is why the recommendation is not "write better documentation" but "move the documentation to where the change happens". Co-location does not make people more conscientious; it makes staleness visible in a diff that somebody is already reading (dbt Concepts).
A wiki page or catalog description per important dataset, holding schema, column meanings, grain, caveats and examples. Updated when someone remembers, reviewed when someone notices it is wrong, and living in a system no code change ever touches.
Column list, types, lineage, compiled SQL, test results and freshness generated from the model on every build. Grain, exclusions, known gaps and intended use written in the model's own definition file, reviewed in the same pull request as the logic, and enforced as tests wherever the claim is checkable.
A documentation claim is only as durable as the mechanism that would break it visibly. Generated content is regenerated from the source of truth and cannot diverge; a written claim beside the code appears in the diff when the logic changes; a written claim in a separate system has no mechanism at all and diverges silently from the first edit onward.
What to write when almost everything is generated
If the column list, types, lineage and test results all arrive automatically, the remaining question is what a human should type — and the answer is short enough to be worth stating exactly: the grain, what is excluded, what is known to be wrong, and what the dataset is for.
The file below is the practical form. The generated page will show the schema, the compiled SQL, the upstream models and the last test run; none of that is typed here. What is typed is the part no machine can produce, and it is mostly caveats.
Note that several claims are simultaneously documentation and tests. unique and not_null on the business key and an accepted-values list on status are assertions a reader can rely on, because a run fails when they stop holding. That is the only mechanism that gives a written claim the durability of a generated one (Data Tests).
- Grain — what one row represents. The single highest-value sentence, and the one that decides whether any aggregate over the table is correct (Grain: What Does One Row Represent?).
- Exclusions — filters applied inside the model. Invisible in the output, and the usual reason two teams get different numbers from the same source (Two Dashboards, Two Numbers).
- Known gaps — late arrivals, periods with different completeness, categories the source stopped emitting. Nothing derives these and consumers cannot infer them (Late-Arriving Data).
- Intended use and non-use — naming the sibling table that has the other grain prevents more misuse than any amount of column prose (Data Products).
- Who owns the definition, separately from who runs the pipeline — because the question a consumer has is usually semantic and the field usually names the operator (Data Ownership).
1version: 22 3models:4 - name: fct_orders5 description: >6 One row per order, as of the latest change record processed. Cancellations7 appear as a status change rather than as a removed row, so every revenue8 measure must filter on status. Internal test accounts are excluded here,9 not downstream.10 meta:11 grain: one row per order_id12 excludes: internal test accounts; orders from the pre-migration system13 known_gaps: >14 refunds arrive well after the order and are written into the original15 order_date partition, so a closed month can change after it closed16 intended_use: revenue and order-volume reporting17 not_for: per-line analysis — use fct_order_lines, which has a different grain18 semantic_owner: finance-analytics19 operational_owner: platform-data20 columns:21 - name: order_id22 description: Business key from the source orders table.23 tests: [unique, not_null]24 - name: revenue25 description: >26 amount_minor converted to the reporting currency at the FX rate as of27 order_date. Gross: before refunds and before tax.28 tests: [not_null]29 - name: status30 description: Order lifecycle state at the time of the last processed change.31 tests:32 - accepted_values:33 values: ['placed', 'paid', 'shipped', 'cancelled', 'refunded']Read what is absent. There is no column list — that is generated. There is no lineage — that is generated. There is no "created_at: the creation timestamp", because a description that restates the column name buries the one about refunds. Everything present here is either a claim no machine can derive, or a claim a machine will now enforce.
Which test types are built in, how custom tests are declared, and how meta fields propagate to a catalog vary by framework and by version. Verify current documentation for the specific keys; the durable idea is that a claim asserted on every run cannot silently become false.
Documentation you can check, and what checking still misses
A documentation claim that is enforced is a different kind of object from a documentation claim that is asserted. The first fails a run when it stops being true; the second waits to mislead somebody. Sorting your claims into those two buckets is the most useful hour a dataset owner can spend.
The checks below are the ones worth running against documentation itself, each with the failure it cannot see. Two of them are purely mechanical drift checks and cost nothing. The interesting one is the last, because it is the only check on this list that requires a person, and it is the only one that catches the failure that actually causes wrong dashboards (Semantic Changes).
The honest conclusion is that documentation quality has a ceiling that tooling cannot raise. Everything structural can be automated; the claim that the description of a metric matches the arithmetic in the model can only be verified by someone who understands both, and no amount of coverage reporting substitutes for that.
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Every column in the schema has a description; every described column exists | The written layer and the generated layer describe the same object. | Columns added without documentation; documentation for columns dropped quarters ago (Schema Evolution). | Descriptions that exist and are wrong, and descriptions that merely restate the column name — both pass, and the second actively dilutes the page. |
| Every published dataset declares a grain | A consumer can tell what one row means before aggregating. | The most consequential omission in analytical documentation, and the most common (Grain: What Does One Row Represent?). | A declared grain that is wrong, or one that was right when written and changed when a join was added — nothing mechanical compares the statement with the model. |
| Uniqueness and not-null on the declared business key, on every run | The declared grain actually holds in the data. | Fan-out joins, non-idempotent re-runs, duplicate delivery — the mechanical breaches of a grain claim (Duplicate Rows). | A grain claim that is wrong in the other direction: rows that should exist and do not. Uniqueness says nothing about completeness (Missing Rows). |
| Accepted-values assertions on enumerated columns | The documented value domain is the real one. | A new status introduced upstream that downstream filters silently exclude (CDC and Schema Drift). | An existing value whose *meaning* changed — the set is unchanged and every row still passes (Semantic Changes). |
| Human review of the definition against the model, on change | The prose and the arithmetic agree. | The gross-versus-net class, the wrong-date-join class, the changed-filter class — the failures that produce wrong dashboards with green pipelines (The Pipeline Succeeded. The Data Is Wrong.). | Everything the reviewer does not know the business meaning of, which makes this check exactly as good as the reviewer and impossible to automate. |
Four of these five are cheap and mechanical and catch structural drift. The fifth is expensive and human and catches the failures that cost money. A documentation programme that implements only the first four will report excellent health right up until a quarter closes on the wrong number.
How to build it
Most important first.
- Generate everything generatable and never write it by hand. Column lists, types, lineage, freshness, test results and the compiled SQL all belong on the page and none of them belongs in a person's editor (Data Lineage).
- Co-locate the written part with the transformation code so a logic change and an unchanged description are visible in the same diff, and reviewable by the same person (dbt Concepts).
- Write grain first, exclusions second, known gaps third. If only three sentences are ever written for a dataset, those are the three (Grain: What Does One Row Represent?).
- Turn claims into tests wherever the claim is checkable. "Order ids are unique" and "status is one of these five values" are documentation that runs, and a failing run is a documentation update request with a stack trace (Data Tests).
- Delete rather than maintain. A column description that duplicates the column name adds nothing and dilutes the caveat that matters; documentation quality is dominated by the ratio of signal to filler, not by coverage.
- Render provenance on the page: which parts were generated and when, which were written and by whom, and how old each written claim is. A reader who can see that a caveat is two years old will treat it correctly (Metadata: Technical, Operational and Business).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Generated sections guarantee they match the model as of the last generation, which for a repository-integrated pipeline is the last merge. That is the strongest documentation guarantee available anywhere.
- Tested claims guarantee they held on the last run of the test, against the data that ran. They do not guarantee they hold for a period the test did not cover, such as a backfilled range (Validating a Backfill Before You Publish).
- Written, untested claims guarantee only authorship and a date. This must be visible in the rendering, because a reader cannot otherwise distinguish a claim the platform enforces from a sentence someone typed.
- No documentation guarantees the reader will use the dataset correctly. Grain stated clearly reduces misuse; it does not prevent someone joining an order-level and a line-level table anyway (Grain: What Does One Row Represent?).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Check for drift between generated and written content: a column present in the schema with no description, and a described column that no longer exists. Both are mechanical, both are cheap, and together they catch the majority of visible rot (Schema Evolution).
- Check that every published dataset states a grain, and that the statement is not a restatement of the table name. The second half needs a human reviewer and is the point at which documentation review earns its place in a pull request.
- Both miss the description that is well-formed, current in appearance, and semantically wrong — the page says gross, the model computes net, every field is populated and every check passes (Semantic Changes).
- Generated sections have the freshness of the pipeline that generates them, which should be every merge. Documentation regenerated nightly is documentation that disagrees with the code for a day after every change.
- Written claims have no freshness unless one is imposed. A review interval with a visible age on each claim is the minimum; a claim older than the interval should be rendered as unverified rather than silently ageing (Metadata: Technical, Operational and Business).
- Tested claims have the freshness of the last test run, which is a real and useful number to display: "asserted on every run, last passed this morning" is a very different statement from "written eighteen months ago" (Data Tests).
- A schema change updates generated documentation automatically and leaves written prose untouched, which means every schema change is also a documentation review whether anyone treats it as one (Schema Evolution).
- A semantic change updates nothing at all: same columns, same types, same description, different meaning. The only mechanisms that catch it are a versioned definition and a human reviewer who knew the old one (Semantic Changes).
- As a dataset gains consumers, its documentation has to shift from "notes for the team that built it" to "an interface description for people you will never meet", and that rewrite is rarely triggered by anything (Data Products).
- Generated documentation is rebuilt by re-running the generator, so it cannot be lost while the repository exists. This is a strong argument for keeping the written part in the same repository rather than in the catalog's database (The Data Catalog).
- Written documentation that lives only in a catalog is recoverable only from that catalog's backups, and organisations discover this during a migration between catalog products.
- When documentation has diverged badly, deleting the stale prose and regenerating is usually better than reconciling claim by claim. An empty section reads as "unknown", which is accurate; a stale section reads as fact, which is not.
What can go wrong
- Documentation in a separate system from the code, so no change to the code ever requires touching it and the divergence is structural rather than cultural.
- A required description field with no reviewer, producing one-word descriptions that restate the column name and dilute the page (The Data Catalog).
- Generated content and written content rendered identically, so a reader cannot tell which claims the platform enforces and which are somebody's recollection.
- Documentation written at dataset creation, when the author understands the dataset least, and never revisited when the logic changed.
- Two authoritative-looking pages for the same model in different systems, disagreeing on the definition, with no mechanism to detect the conflict (The Metrics Layer).
- "Documentation coverage is the goal." Coverage measures how many fields are non-empty. A page of column descriptions that restate column names has full coverage and negative value, because it buries the one sentence about refunds (The Data Catalog).
- "Generated documentation is enough." It is true and it cannot state grain, exclusions or intent — the three things that decide whether a consumer uses the dataset correctly (Grain: What Does One Row Represent?).
- "The description is in the catalog, so it is maintained." Being in the catalog is what makes it unmaintained: no change to the model requires touching it. Location is the mechanism, not the storage (dbt Concepts).
- "We should document everything." Document what cannot be derived and enforce what can be tested. Everything else dilutes (Data Tests).
- Column-level descriptions are where classification decisions get justified. A column marked as personal data with no description of what it holds cannot be assessed for minimisation or retention by anyone except its author (Data Classification).
- Documented intended use is the only artefact that makes purpose limitation reviewable: a dataset assembled for fraud detection and quietly reused for marketing is a governance event, and the documentation is what makes it visible (Data Minimization).
- Known-gaps documentation has a compliance dimension. A dataset used in a regulated report whose known incompleteness is undocumented transfers a real risk to every consumer downstream (Data Retention).
Operating it
- Share of published datasets with a stated grain, and share of columns with a description that is not a restatement of the column name. The second requires sampling and is worth it.
- Age distribution of written claims, and the count past the review interval (Metadata: Technical, Operational and Business).
- Count of documented columns that no longer exist and undocumented columns that do — the drift number, in both directions (Schema Evolution).
- Share of documented claims that are also enforced as tests. This is the ratio that says how much of the documentation is durable (Data Tests).
- At ten times the model count, hand-written documentation stops being maintainable and the ratio of generated to written content decides whether the platform is navigable at all.
- At a hundred times, only tested and generated claims survive; everything else is treated by readers as historical. Planning for that reality early is what stops a documentation programme becoming a debt-repayment programme.
- Consumer count changes what documentation must contain rather than how much. One team needs notes; forty consumers need an interface description with a grain, exclusions, an SLO and a deprecation policy (Data Contracts).
- Generation is close to free and runs with the transformation build. The cost is entirely the human time to write and review the small non-derivable part, which is why keeping that part small is an economic decision (What Actually Drives Data Platform Cost).
- Tests that encode documentation cost compute on every run, and the expensive ones are the whole-table assertions — uniqueness across a large fact table is a full scan and should be argued for on the same terms as any other scan (Scan Cost).
- The cost avoided is a misused dataset. Two teams computing revenue differently because neither page stated the exclusions is a recurring, invisible cost that never appears as an incident (Two Dashboards, Two Numbers).
- Co-locating documentation with transformation code keeps it honest and puts it behind a pull request, which excludes the business people who often know the definitions best. Some organisations solve this with review rather than access, and some simply lose those contributions.
- Turning claims into tests makes them durable and makes every claim a potential pipeline failure. A test on a claim nobody would act on is a future 03:00 page for a cosmetic assertion (Quality Alerting).
- Generating aggressively produces pages that are always true and mostly uninteresting. The judgement call is how much filler to allow before the caveat that matters gets skimmed past.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe split between derivable and non-derivable claims holds for any stack, because it follows from whether a fact changes when the code changes rather than from any tool. What varies is how much a given framework generates automatically, which shifts the boundary but never removes it.
- TOOL-SPECIFICTransformation frameworks that keep model definitions and descriptions in the same repository make co-location the default, while catalog-first workflows put the description in a separate database where no code change touches it. The same team produces very different documentation decay depending only on which of the two they adopted.
- ORG-SPECIFICWhether documentation must be an interface description or can remain team notes depends on how far the consumer set reaches. Within one team, shared context substitutes for most of it; across organisational boundaries, the unwritten context is exactly what produces two different revenue numbers.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the delivery mechanics that make co-location work — documentation generated and published by the same pipeline that deploys the models, so a merge updates both or neither.