MetadataGENERALTOOL-SPECIFICORG-SPECIFIC

Metadata: Technical, Operational and Business

Schema, owner, description, freshness, lineage, tags and quality — split by where each comes from, because that is what predicts which of them is still true.

What actually happens How to build it Can I trust it?

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

What do you have to know about a dataset before you are willing to put its numbers in front of someone who will act on them?

Who needs this

Four people with four different questions: an analyst deciding whether fct_orders is the right table at all; an on-call engineer at 03:00 deciding whom to wake; a privacy reviewer asking which columns hold personal data; a producer deciding whether tomorrow's schema change is safe to ship. All four give up in about a minute if the answer is not in one place, and then guess.

What one row is

One metadata record is one assertion about one object — a table, a column, a job, a run, a dashboard. The object is the grain, and mixing levels is the classic mistake: freshness belongs to a table, nullability belongs to a column, duration belongs to a run. A store that flattens all three into "table properties" can never answer a column-level question, which is the question that matters during a privacy review or an incident (Column-Level Lineage).

The obvious build

A wiki page — or a spreadsheet — listing the important tables with a description and an owner. Somebody senior writes it in an afternoon, it is genuinely the most useful artefact on the platform for a quarter, and it costs nothing to start. Almost every catalog in existence began as this and it was the right first move.

Why it breaks

The page is written once and the platform keeps moving. Six months later a third of the entries describe tables that no longer exist, and the real tables added since are missing. Nothing marks which third is stale, so the whole page loses credibility at the rate of its worst entry.

How it breaks with real data

The page is written once and the platform keeps moving. Six months later a third of the entries describe tables that no longer exist, and the real tables added since are missing. Nothing marks which third is stale, so the whole page loses credibility at the rate of its worst entry.
The owner cell names a person who left the company. Nobody noticed, because the only moment anyone reads that cell is during an incident — the worst possible moment to discover it (Data Ownership).
Freshness is recorded as "daily". That was a statement of intent when it was typed and it is still a statement of intent now; the actual answer changes every day and no human is going to retype it (Freshness Monitoring).
A column is renamed upstream. The wiki shows the old name, the warehouse shows the new one, and there are now two sources of truth about a schema — which is strictly worse than one, because a reader has to decide which to believe (Schema Evolution).
"Which tables contain an email address?" arrives before a privacy review. The page holds table-level prose and the question is column-level, so the honest answer is a week of grepping (PII in Pipelines).

Source→Ingestion→Raw→Transformation→Validation→Storage Model→Serving→Consumer→Observability

What is actually happening

Metadata splits cleanly by where it comes from, and the source predicts its half-life better than any process you can put around it. Technical metadata — table and column names, types, nullability, partition keys, file counts, table format version — is *harvested* from the system that already holds it. It cannot drift, because it is re-read from the authority on every harvest.
Operational metadata — last run time, rows read and written, bytes scanned, run duration, the job and code version that produced this table — is *emitted by runs*. It is accurate for the same reason a log is accurate: it is a by-product of execution and nobody has to remember to write it (Pipeline Metrics).
Business metadata — what one row represents, which definition of "active customer" this uses, what the known gaps are, who owns it, what it must not be used for — exists nowhere in any machine. Only a human can write it. This is simultaneously the most valuable third and the only third that rots (Dataset Documentation).
The rot is structural, not a discipline failure. Harvested and emitted metadata are re-derived continuously, so their accuracy is restored on every cycle. Human-written metadata is derived once, so its accuracy can only decay from the moment it is saved. A catalog that renders all three with the same visual weight is actively misleading about which parts are still true.
The counter is to shrink the human-written surface, and to bind whatever survives to something that changes when the data changes — the description lives beside the transformation that produces the column, in the same commit, reviewed in the same diff (dbt Concepts).
Everything else in this module is a consequence. A catalog is a place to put harvested metadata (The Data Catalog); lineage is operational metadata emitted by transformations (Data Lineage); discovery is ranking over it (Data Discovery); ownership is the one business field you cannot do without (Data Ownership).

Three kinds of metadata, and only one of them rots

GENERALThe classification is stable across tools, but the boundary moves: pattern scanners can propose classification tags that used to be purely human, and some engines emit lineage that elsewhere has to be parsed. The direction of travel is always from authored towards harvested, never back.

Ask what metadata a dataset needs and you will get a flat list: schema, owner, description, freshness, lineage, tags, quality status. The list is right and the flatness is the problem, because those seven fields have three completely different lifecycles and the flat list hides that.

Sort them instead by who can produce the fact. If the source system already knows it, the answer is to read it — repeatedly, forever. If a run can emit it as a side effect, the answer is to instrument the run. If neither is true, a person has to type it, and from that moment its accuracy can only fall.

This is not a taxonomy for its own sake. It tells you where to spend engineering effort (on harvesters and emitters), where to spend social effort (on the small authored surface), and how to render the result — because a value that was harvested an hour ago and a value a person typed two years ago must not look the same on screen.

Field	Kind	Produced by	Half-life	What it still cannot tell you
Table and column names, types, nullability	Technical	Harvest from `information_schema` or the table format's manifest	Always current, up to the harvest interval	What any of it means. `status VARCHAR` is a shape, not a definition.
Partition keys, file count, table size	Technical	Harvest from the catalog or storage layer	Always current	Whether the layout suits the queries people actually run (Partition Pruning).
Last successful run, rows written, duration	Operational	Emitted by the job at the end of each run	As fresh as the last run — and misleading once runs stop	Whether the rows written were correct; a green run writing wrong values emits an identical record.
Lineage edges: which job read what and wrote what	Operational	Emitted by the transformation tool, or parsed from the SQL it ran	Current for instrumented paths; blind elsewhere	Anything that moved data outside the instrumented tools — an export, a notebook, a hand-run script.
Freshness against SLO	Operational	Computed from the newest complete record versus now	Continuously recomputed	Whether fresh data is right. Freshness and correctness are independent (Freshness Checks).
Owner	Business	A human, once, at registration	Decays with team reorganisations and departures	Whether that team agrees they own it. Assigned ownership and accepted ownership are different things.
Description, grain, known gaps, intended use	Business	A human who understood the dataset at the time	Decays from the moment it is saved	Nothing — this is where the actual value is, and it is the part with no mechanism keeping it true.
Tags and classification	Business, sometimes inferred	Assigned by a human, or proposed by a scanner and confirmed	Decays as new columns arrive untagged	Whether a newly added column carrying personal data was ever seen by the scanner (Data Classification).

The path a single metadata fact takes

Metadata has a pipeline exactly like the data it describes, and it fails in the same ways: a stage that stops running, a stage that publishes partially, a stage whose output nobody validates. Treating it as a pipeline rather than as a feature of a product is what makes it debuggable.

Read the guarantee column below. Notice that no stage promises the fact is *true* — the strongest promise anywhere in the chain is that a value was read from an authority at a stated time, and that promise weakens with every hop just as it does in the data plane (The Fundamental Data Journey).

The stage most often missing is the last one. Plenty of platforms harvest diligently and then present the result without any indication of age, which converts an honest snapshot into an implied live view.

From a fact in a source system to a fact on a catalog page

1
Harvester
Connects to each source and reads schemas, partitions, object listings and table statistics.
guarantees The values it returns were true in the source at the moment of reading, and it records that moment.
fails by Failing silently on one source out of twelve, leaving that source's section of the catalog frozen at its last successful run while everything around it updates.
2
Run emitter
The transformation job reports its run id, code version, inputs read, output written and row counts.
guarantees A record exists for every run that reached the emit point.
fails by Emitting only on success, so the catalog's view of the platform excludes every failure — which is the set an investigator is looking for.
3
Authoring
A human writes grain, meaning, known gaps and intended use, ideally in the transformation repository.
guarantees Someone believed this at the time of writing. Nothing more.
fails by Being written once at dataset creation, when the author understands it least, and never revisited when the logic changes underneath it.
4
Merge and store
Reconciles all three origins into one record per object, keyed by a stable identifier.
guarantees One record per object, with each field carrying the origin and timestamp it came from.
fails by Keying on a mutable name, so a renamed table appears as a brand-new object and its authored metadata is orphaned on the old one.
5
Index
Builds the search structures behind discovery — names, descriptions, tags, usage counts.
guarantees Objects present at index time are findable by the fields that were indexed.
fails by Lagging the store, so a table that exists and is catalogued is still unfindable and a user concludes the catalog does not have it.
6
Presentation
Renders the record to a human who is about to make a decision on it.
guarantees Nothing about truth. It can only guarantee it displays each field's origin and age honestly.
fails by Rendering a two-year-old description and an hour-old row count with identical styling, so the reader trusts both equally.

Every stage here is a data pipeline stage and deserves the same treatment: monitoring, freshness, and a check on the far end. A catalog that nobody monitors goes stale exactly as quietly as a table that nobody monitors.

The catalog is a dataset, so check it like one

The fastest way to find out whether a metadata layer is real is to reconcile it against the system it claims to describe — the same move as reconciling a warehouse table against its source, and for the same reason (Reconciliation).

The query below is deliberately dull. It full-outer-joins what the warehouse actually contains against what the catalog thinks it contains, and buckets the result. Every bucket except ok is a different organisational problem: uncatalogued means the platform is growing faster than the harvester, ghost means the harvester is not seeing deletions, unowned means the registration process has an optional field that should not be optional.

What the query cannot do is validate meaning, which is why the checks table below it names the blind spot for each check rather than only the coverage. The pattern generalises: every check on metadata catches a structural error and misses every semantic one.

Checks on the metadata layer itself, and what each still misses

Check	Expresses	Catches	Still misses
Catalog coverage against live objects, both directions	The catalog describes the platform that exists, not the one that existed at launch.	Tables created outside the registration path; tables dropped without the catalog noticing; a harvester that has been failing on one source.	Everything about whether the catalogued entries are useful. A fully covered catalog of meaningless descriptions passes this check perfectly.
Live schema versus catalogued schema, column by column	What the catalog says about shape matches what the engine will return.	Stale harvests; upstream changes nobody announced; a column dropped between harvests (Breaking Schema Changes).	Semantic change with an unchanged schema — the switch from gross to net that leaves every type identical (Semantic Changes).
Datasets with no owner, or an owner that resolves to no active team	Every dataset has somebody who can be asked and can be paged.	Departures, reorganisations, and datasets registered with the field left blank.	Ownership that is assigned but not accepted — the team is real, is listed, and has never agreed to answer for this table (Data Ownership).
Description age against a review interval	Somebody has looked at this claim recently enough to be worth believing.	Descriptions written at creation and abandoned; datasets whose logic changed after the last review.	A recently reviewed description that is confidently wrong. Review date measures attention, never accuracy.
Lineage edge count per dataset compared with its own history	The instrumented paths are still emitting.	A transformation tool that stopped emitting after an upgrade; a job moved outside the instrumented framework (Data Lineage).	Data movement that was never instrumented at all — a CSV export, a notebook, a hand-run script. Absence of an edge is not evidence of absence of a dependency.

Read the misses column top to bottom: every one of them is semantic. Structural checks on metadata are cheap and worth having; not one of them substitutes for a person who knows what the dataset means.

Catalog coverage: what exists versus what is catalogued

1-- :stale_before is a date the organisation chooses — the age past which a
2-- human-written description is displayed as unverified rather than as fact.
3with live as (
4  select table_schema, table_name
5  from information_schema.tables
6  where table_type = 'BASE TABLE'
7),
8known as (
9  select table_schema, table_name, owner_team, described_at
10  from catalog_tables
11)
12select
13  coalesce(l.table_schema, k.table_schema) as table_schema,
14  coalesce(l.table_name,   k.table_name)   as table_name,
15  case
16    when k.table_name is null                        then 'uncatalogued'
17    when l.table_name is null                        then 'ghost'
18    when k.owner_team is null                        then 'unowned'
19    when k.described_at is null                      then 'undescribed'
20    when k.described_at < :stale_before              then 'description unverified'
21    else 'ok'
22  end as status
23from live l
24full outer join known k
25  on  l.table_schema = k.table_schema
26  and l.table_name   = k.table_name
27order by status, table_schema, table_name

The full outer join is the whole point: an inner join would report only on tables both sides already agree about, which is exactly the population that has no problem. The two anti-joins — rows present on only one side — are where every finding lives.

Product detail — verify current documentation

Catalog products differ substantially in what they harvest without configuration — some read warehouse schemas continuously and treat object storage as opaque, others do the reverse. Verify current documentation for what a given product harvests automatically before assuming any of the fields above arrive for free.

How to build it

Most important first.

Harvest everything harvestable. Any field a human is asked to type that a system already knows is a field that will eventually disagree with the system, and the human will lose (The Data Catalog).
Emit operational metadata from the job itself: run id, code version, inputs read, output written, rows in and rows out. That last pair is simultaneously operational metadata and the cheapest data-quality signal available (Volume Anomalies).
Keep business metadata next to the code that produces the dataset. Then changing the logic without changing the description is a visible diff in review rather than an invisible omission (Dataset Documentation).
Make ownership required and make its value a team, not a person. People leave at a much higher rate than teams are dissolved, and a team has a rota behind it (Who Owns Data Quality).
Attach a freshness to every metadata field, not just to the data. "Owner: payments-data, asserted fourteen months ago" is a far more useful cell than "Owner: payments-data", because it lets a reader discount it correctly.
Decide what you will deliberately not record. A model with forty optional attributes produces a form nobody completes; three required fields that are always populated beat twenty that are usually empty.

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

Harvested technical metadata is accurate as of its last harvest and no more. It is a timestamped snapshot, not a live view, and during a migration the harvest interval is a real window in which the catalog is confidently wrong.
Operational metadata is accurate for runs that emitted it and silent about runs that died before emitting. The absence of a run record is ambiguous between "did not start" and "started and was killed", and those two need opposite responses (When a Task Fails Mid-DAG).
Business metadata guarantees only that somebody once believed it. No mechanism anywhere makes a description true, so a description that contradicts the data is a normal state rather than an anomaly.
Nothing guarantees the completeness of the catalog itself. The datasets it does not know about are precisely the ones nobody registered, which correlates almost perfectly with the ones nobody owns.

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this

The highest-value check is coverage against reality: enumerate every table the warehouse actually holds, compare with the set the catalog knows, and alert in both directions. Tables the catalog has never heard of are a governance gap; entries pointing at tables that no longer exist are a credibility gap, and credibility is the whole product here.
A second check diffs each catalogued schema against the live schema, column by column. A mismatch is either a stale harvest or a change nobody announced, and both are worth a notification (Breaking Schema Changes).
Both miss the entire business layer. Nothing can detect that a description says "gross revenue" while the model computes net — the only comparator that could is a person who understands both, and no scheduled job can be that person (Semantic Changes).

Freshness

The harvest interval decides how long a new table stays invisible and a dropped one stays listed. Both are read by users as evidence about the platform, so a slow harvester makes the catalog feel wrong even when every value it holds is right.
Operational metadata is as fresh as the last run — which means a pipeline that stopped entirely presents as "fresh as of its last run" forever, unless the catalog compares that timestamp against the dataset's declared schedule (The Freshness SLO).
Business metadata has no natural freshness at all. You have to impose one: a review date, or a rule that renders any description older than some agreed age as unverified rather than as fact.

When the schema or meaning changes

A source schema change must arrive in the catalog as a *change event*, not merely as a new snapshot. "The schema is X" tells a consumer nothing; "column amount_cents disappeared on Tuesday and amount_minor appeared" is the fact they need (Schema Evolution).
The metadata model itself evolves, and it evolves in the direction of more fields. Every field added is a field that will be empty for the existing thousand datasets, which is why backfilling metadata is its own small data project.
Meaning changes without the schema changing at all, and the metadata layer has no way to notice. This is the failure the evolution field exists to keep visible: a column that switched from gross to net has an unchanged type, an unchanged name, and an unchanged description (Semantic Changes).

How to re-run this safely

Harvested metadata is trivially recoverable — drop it and re-harvest, because the authority still exists. This is worth stating because it means the harvest store can be treated as a cache, with all the operational freedom that implies.
Operational metadata is recoverable only for the retention window of the run history that produced it. Once run records age out, "which code version produced this partition" becomes unanswerable, which is exactly the question a backfill investigation asks (Validating a Backfill Before You Publish).
Human-written metadata is not recoverable by any mechanism. If it lives only in a catalog's database and that database is lost, it is gone — which is the strongest argument for keeping descriptions in the transformation repository, where they are versioned alongside everything else.

What can go wrong

Failure modes

The catalog is populated once at launch, is never re-harvested, and drifts into a confident, authoritative-looking record of a platform that no longer exists.
Harvesting is aggressive enough to be noticed — a metadata crawler issuing schema queries against every source on a tight loop can compete with the production workload it is describing (Workload Isolation).
Operational metadata is emitted only on success, so the catalog's picture of the platform silently excludes every failure — the exact set of runs an investigator wants (Pipeline Observability).
Ownership is recorded as a person, the person leaves, and the field is now a name that resolves to nobody but still looks filled in.
The metadata store becomes a second production dependency: if the catalog is down, deploys are blocked because a policy requires a catalog lookup, and a documentation system has become an availability risk.

Misreads

"We bought a catalog, so we have metadata." You have a place to put it. The harvested third arrives free; the third that answers "should I use this table" still has to be written by someone who knows, and the tool has no opinion about whether they did (The Data Catalog).
"Metadata is documentation." Two of the three kinds are operational data with a freshness SLO, and treating them as documentation is what leads to their being generated once and never again.
"The schema is the metadata." The schema tells you the shape and nothing about the meaning. amount DECIMAL(12,2) is compatible with gross, net, per-line, per-order, in any currency, tax-inclusive or not (Grain: What Does One Row Represent?).
"More fields make a better catalog." Completion rate falls faster than field count rises, and a form that is half empty teaches users that the catalog is half wrong.

Privacy, retention and access

Column-level metadata is the substrate for every privacy control that follows. Classification, masking policies, retention rules and access grants are all expressed against columns, so an organisation without column-level metadata cannot enforce them except by hand (Data Classification).
Metadata itself can be sensitive. Column names, sample values and query history leak business structure and sometimes personal data — a catalog showing "top values" for a column called patient_diagnosis has become a disclosure channel (Data Access Control).
Retention applies to operational metadata too. Run histories and query logs describe who read what and when, which is useful for discovery and is also a surveillance record with its own retention obligation (Data Retention).

Operating it

How you see it in production

Catalog coverage as a percentage of live tables, trended over time. The trend matters more than the value — coverage falling means the platform is growing faster than the harvester or the culture around it.
Time since last harvest per source, exposed in the UI next to every value that came from it, so a reader can see the age of what they are being shown.
Count of datasets with no owner, and count with a description older than the review interval. These two numbers are the honest health check on the human-written third (Data Ownership).
Metadata write volume by origin — harvested, emitted, authored. A platform where the authored share is growing is a platform that has decided humans should do a machine's job.

What changes at 10x and 100x

At ten times the dataset count, the catalog stops being a browsable list and becomes a search problem. Nobody scrolls a thousand tables, so ranking is now the product (Data Discovery).
At a hundred times, the metadata store is itself a data system with a schema, a freshness SLO and an incident history. Treating it as a side project at that point is how a platform ends up with a catalog nobody trusts.
Consumer count scales the governance question rather than the technical one. Ten datasets and three analysts need no catalog at all; five hundred datasets and eighty consumers cannot function without one, and the transition between those states happens without anyone announcing it.

What drives cost here

The dominant cost is harvest traffic against sources: listing objects in a bucket, querying information_schema, describing partitions. It grows with the number of objects, not with their size, so a lake full of small files is expensive to catalog for the same reason it is expensive to query (File Size and the Small-Files Problem).
Operational metadata volume scales with runs times datasets touched, which grows fastest in platforms with many small models — a fine-grained transformation DAG produces far more lineage events than a coarse one (The Transformation DAG).
The largest cost is not computational at all. It is the human time spent writing and reviewing business metadata, and it is the reason the harvest-first design is an economic argument rather than an aesthetic one.

What this approach costs

Harvest-first gives you metadata that is always true and rarely interesting. Names, types and row counts do not tell an analyst which of three revenue tables to use; that answer is only ever human-written, so you cannot escape the rotting third — only shrink it.
Binding descriptions to transformation code makes them accurate and makes them a developer artefact. Analysts who know the business best now need a pull request to fix a description, and some of them will simply not do it (Dataset Documentation).
Requiring ownership before a dataset can be published raises quality and slows everyone down. It is the right trade above a certain team count and pure friction below it.

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

GENERALThe three-way split by origin — harvested, emitted, authored — holds for every metadata system regardless of product, because it follows from who can produce each fact rather than from any tool's design. What differs is how much of each a given tool collects automatically.
TOOL-SPECIFICHow much is harvested for free varies enormously: a warehouse-native catalog reads schemas continuously and knows nothing about your files, while a lake catalog knows about files and needs an engine to tell it about types. Assume nothing about coverage until you have run the coverage check against reality.
ORG-SPECIFICWhether business metadata is worth requiring depends on team count, not data volume. In a single-team company the metadata lives in the team's head and a catalog is ceremony; across ten teams the same absence produces daily misuse of datasets nobody can explain.

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

Databaserequirements-to-schema sql-basics normalization

Backendstructured-logging

API Designapi-documentation api-ownership

Securitydata-classification audit-logs

Observabilitywhat-observability-is instrumentation

Domains that do not exist yet

— DevOps / Production Engineering owns the delivery side of this: how a description that lives in the transformation repository gets reviewed, versioned and shipped with the model it describes. That domain is being built separately.
— Distributed Systems owns why a harvested snapshot from many independent sources can never be a consistent view of all of them at one instant — the catalog is a distributed read, with all that implies.