Data Governance
Ownership, classification, access, retention and auditability as five mechanisms with enforcement points — not as a document in a wiki.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
For any table in the warehouse, can you answer who owns it, what class of data it holds, who may read it, how long it is allowed to exist, and who read it last week?
Three audiences that never talk to each other. A security reviewer wants to know which datasets hold regulated fields and who can reach them. A legal or privacy function wants retention and deletion to be real rather than aspirational. And an analyst wants access to a table today, without a two-week ticket, which is the audience governance programmes forget and the reason they get routed around.
The governed unit is one column of one dataset, not one dataset. Classification, masking and retention all attach at column granularity, because a table is almost never uniformly sensitive: fct_orders holds an order id nobody cares about and a delivery address that is regulated. A table-level policy is the maximum of its columns applied to all of them, which is why table-level governance is simultaneously too strict to be usable and too coarse to be safe.
Write a data governance policy. It defines classification tiers, says PII must be protected, assigns a data owner per domain, and lives in the company wiki. It is reviewed annually. This is not stupid — you genuinely do need the definitions written down, and every mechanism below refers back to them.
The policy governs datasets that a human created. Pipelines create datasets: every dbt run materialises staging models, every notebook writes a scratch table, every ingestion job lands a new source table the week a SaaS tool is bought. The wiki lists forty datasets; the warehouse holds four thousand (Data Discovery).
- The policy governs datasets that a human created. Pipelines create datasets: every
dbt runmaterialises staging models, every notebook writes a scratch table, every ingestion job lands a new source table the week a SaaS tool is bought. The wiki lists forty datasets; the warehouse holds four thousand (Data Discovery). - Classification is assigned to the table at creation and never re-evaluated. Then a transformation joins a customer dimension onto an event table, and a dataset that was classified
internalnow carries email addresses, with its original label intact (Data Classification). - The "data owner" named in the wiki left the company. Nothing broke, because the owner was never a mechanism — no grant, no alert and no approval routed through them (Data Ownership).
- Retention says raw data is kept for a bounded period. Object storage lifecycle rules were configured on one prefix, and the ingestion job started writing to a new prefix eight months ago (Storage Lifecycle).
- An auditor asks who accessed a sensitive table in March. The warehouse has query history, the lake has bucket access logs with a shorter horizon, the BI tool has its own log, and the notebook that pulled a CSV to a laptop has none of them (Audit Logs for Privileged Actions).
- Because access requests take two weeks, an analyst who needed one column got a full extract from a colleague over Slack. The governed path was slower than the ungoverned one, and governance lost on latency (The Self-Service Data Platform).
What is actually happening
- Governance is five separable mechanisms that people bundle into one word. Declaration: metadata saying what this dataset is, who owns it and what class it holds. Enforcement: a system that actually denies a read. Propagation: lineage that carries a declaration from a source column to every derived column. Expiry: a process that deletes bytes on a clock. Evidence: a log proving who did what. A programme that has the first and the fifth but not the middle three is documentation, not governance.
- The declaration lives in a catalog, and its only real property is that it is queryable — you can ask "which datasets contain a column classified as personal data" and get an answer that is not a spreadsheet (The Data Catalog, Metadata: Technical, Operational and Business).
- Enforcement lives in whichever system holds the bytes, and there are always several: the warehouse has grants and policies, the object store has bucket-level IAM, the catalog may have its own layer, and the orchestrator runs as a principal that can read all of it (Data Access Control).
- Propagation is the part with no manual equivalent. A human can classify a hundred source columns. Nobody can classify the ten thousand derived columns that inherit from them, so classification either propagates along column-level lineage or it is wrong within a quarter (Column-Level Lineage).
- Expiry is a job, and like every job it can fail silently. Retention that is written in a policy and not expressed as a lifecycle rule, a partition-drop task or a table TTL is a statement of intent about bytes that are still sitting there (Data Retention).
- Evidence is access logging, and its horizon is usually shorter than the questions asked of it. The audit question always arrives months after the access (Audit Trails).
The five mechanisms, and which one your organisation skipped
A governance programme is easy to assess if you stop reading its documents and instead ask, for one specific table, five questions in order: what does the platform *say* about it, what *denies* a read of it, how did that declaration reach every table derived from it, what deletes it, and what proves who read it. Almost every organisation can answer the first and the last and stalls on the middle three.
The reason is that declaration and evidence are additive — you can bolt a catalog and an access log onto an existing platform without changing any pipeline. Enforcement, propagation and expiry are invasive: they change how tables are created, how transformations are written and how storage is laid out. The cheap parts get done first, and the resulting programme looks complete from the outside.
The stage table below is a governed dataset's lifecycle rather than a data flow, and the guarantees column is the point. Read it as a chain: each stage can only promise something if the one before it did.
- 1Declare
Registers the dataset in the catalog with an owner, a purpose, a grain and a per-column classification.
guarantees That a statement about this dataset exists and is queryable by other systems. Nothing about its truth.
fails by Being optional. A pipeline that can publish without registering produces datasets that no later mechanism will ever see.
- 2Classify
Assigns each column a class, by declaration for sources and by inheritance for derived columns.
guarantees Coverage only as far as lineage can be resolved; a column produced by an unparsed transformation is unclassified, not safely classified.
fails by Defaulting unknown to the lowest class, which turns every lineage gap into a silent downgrade.
- 3Enforce
Grants, row filters and column masks evaluated by whichever system serves the bytes.
guarantees That reads through that specific system obey the policy. Never that other paths to the same bytes do.
fails by Covering the warehouse and not the object store the warehouse reads from.
- 4Serve
Analysts, BI tools, notebooks and models read the data.
guarantees Nothing whatsoever. Every read here is a potential new, ungoverned copy.
fails by A CSV export, which converts a governed dataset into an ungoverned one in a single click.
- 5Expire
Deletes partitions, objects or rows on a retention clock.
guarantees That bytes under the paths and tables it actually targets are removed on schedule.
fails by Running successfully against a prefix the pipeline stopped writing to eight months ago.
- 6Prove
Retains access and change logs, and answers "who read this, when".
guarantees Answers only for surfaces that log, and only back as far as the shortest log retention.
fails by Being asked a question older than its own retention, which is the normal case.
The Serve stage guaranteeing nothing is not a gap to be fixed — it is the honest statement of where platform mechanisms end and organisational ones begin. Every control before it exists to make that stage smaller.
Where a policy is actually enforced
Draw the paths to your data and you will find more than you expected. The warehouse is one. The object store underneath it is another, and external tables mean the same bytes are reachable both ways. The orchestrator reads everything by construction. The BI tool caches extracts. Backups are a full copy with their own access model. Each of those is an enforcement point, and the effective policy is the weakest one.
This is the single most useful diagram in governance because it reframes the question. The question is not "have we configured permissions" — it is "how many doors are there, and is anyone watching the ones that are not the front".
The pattern to aim for is fewer doors: the object store readable only by the warehouse and the processing engines, humans reaching data only through a system that evaluates policy, and the orchestrator's identity scoped per pipeline rather than shared (Least Privilege in Infrastructure).
Proving it, and what the proof still misses
Governance claims are testable, and the tests are ordinary scheduled queries over catalog and system metadata. Treating them as data tests rather than as an annual review is what turns a programme from a document into a mechanism, because a failing test has an owner and a deadline while a review finding has neither (Data Tests).
Each check below has a blind spot, and the blind spots share a shape: they are all bounded by what the catalog knows. Coverage is therefore the check that must exist before the others mean anything — a platform where governance checks pass on the twelve per cent of datasets that are registered is not a governed platform, it is a well-tested sample of one.
Note the last row. Access logging is the only check here that can be satisfied and still be useless, because the audit question always arrives after the log horizon has passed. Log retention is a governance decision, and it is usually made by whoever configured the tool.
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Every warehouse dataset appears in the catalog | Coverage — the ceiling on every other governance guarantee. | Pipelines publishing datasets outside the registration path; ad-hoc tables created by humans and never removed. | Data that never reaches the warehouse at all — bucket prefixes, message-log topics, and extracts already outside the platform. |
| Every dataset has an owner who is a current employee | That someone can be asked a question about this data and can approve access to it. | Ownership decay after reorganisations and departures, which is the normal way ownership dies. | An owner who exists but does not know they own it, which is functionally identical to no owner and structurally invisible. |
| No column is unclassified downstream of a classified column | Classification propagated along lineage rather than stopping at a parser gap. | Personal data flowing into a model whose output inherited a lower class; unparsed transformations creating silent downgrades. | A column classified confidently and wrongly at the source, which propagates its error perfectly. |
| Oldest object under each retention-governed path is within its stated horizon | That expiry is running against the paths currently being written. | A lifecycle rule attached to a prefix the writer moved away from; a failing retention job nobody alerts on. | Copies outside the governed paths — snapshots, backups, cross-region replicas and the tables a backfill wrote to a scratch schema. |
| Every read of a sensitive dataset appears in an access log retained beyond the audit horizon | Evidence — that the question "who read this in March" is answerable in September. | Surfaces with no logging at all, which is the common gap for lake reads and BI extracts. | Everything after the copy leaves the platform, and any access whose log has already aged out. A log with a short horizon satisfies the check and fails the audit. |
Run these as data tests with owners and alerts, not as an annual questionnaire. The difference between a control and a claim is whether something fails when it stops being true.
How to build it
Most important first.
- Make the catalog the declaration point and make registration a precondition for publishing, not a follow-up task. A dataset that a pipeline can create without an owner and a classification is a dataset that will exist unowned for years (Data Ownership).
- Default-deny for new datasets. The failure mode of default-allow is invisible — nobody notices they can read something they should not — while the failure mode of default-deny is a ticket, which is loud and gets fixed (Fail Open vs Fail Closed).
- Attach policy to classification, not to table names.
mask every column tagged personal-emailsurvives a table rename, a schema refactor and a new dataset;mask fct_customers.emailsurvives none of them (ABAC and Policy-Based Authorization). - Give the fast path governance rather than giving governance a fast path. Self-service access to classified-low data with automatic approval removes the incentive to route around the system, which is what actually leaks data (The Self-Service Data Platform).
- Express retention as a mechanism in the same repository as the pipeline that writes the data, so that changing where a job writes and changing what expires it are the same review (Storage Lifecycle).
- Log access at every surface that can produce a copy, and treat the surface with the shortest log horizon as the platform's real audit horizon (Audit Logs for Privileged Actions).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- A catalog guarantees that a declaration exists and is retrievable. It guarantees nothing about whether the declaration is true, and nothing at all about enforcement — a catalog with no policy engine behind it denies no reads.
- A warehouse grant guarantees that queries through that warehouse are subject to it. It says nothing about the object-store path underneath an external table, the snapshot in the backup bucket, or the extract in a BI tool's cache (Data Access Control).
- Column-level lineage guarantees propagation only for transformations it can parse. A model written in Python, a UDF, or SQL the parser does not understand produces a lineage gap, and classification stops there silently (Column-Level Lineage).
- Nothing here guarantees that data outside the platform is governed. Once a row is in a spreadsheet, the platform's mechanisms have ended and only organisational ones remain.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The posture check: for every dataset in the catalog, assert that it has a named owner who is still an employee, a classification, a retention rule and at least one access grant that is not the pipeline's own service account. Run it as a scheduled query over catalog metadata and treat failures as data incidents.
- It misses everything the catalog does not know about — the bucket prefix nobody registered, the personal project dataset, the extract on a laptop. Coverage of the catalog is the ceiling on the value of every governance check that reads it.
- It also cannot tell you that a classification is *correct*. A column labelled
internalthat holds free-text support tickets full of personal data passes every structural check ever written (Data Classification).
- Governance metadata has its own freshness, and it is the one nobody monitors. A classification derived from a scan run last quarter describes last quarter's schema.
- The gap that matters is between a dataset being created and being governed. If a pipeline can materialise a table at 03:00 and the classification scan runs weekly, there is a window measured in days where sensitive data exists with no policy attached to it.
- Access revocation has a freshness too. A grant removed in the identity provider is not necessarily a session terminated in the warehouse, and a long-running query or a cached BI extract can outlive the revocation (Sessions).
- Schemas change and classifications do not follow them automatically. The specific dangerous case is a column added upstream that lands in a permissive
SELECT *staging model and inherits the model's existing, lower classification (Schema Evolution). - Meaning changes without schema changes. A
notescolumn that held delivery instructions and now holds customer-service transcripts is a reclassification event with no DDL to trigger it (Semantic Changes). - Regulatory scope changes too, and it changes retroactively over data you already hold. This is the argument for expressing classification as tags evaluated by policy at query time rather than as a physical split of tables — retagging is cheap, re-partitioning a lake is not.
- Recovering from over-exposure is partly impossible: you can revoke a grant, but you cannot un-read a row. The recoverable part is the surface — revoke, rotate any credential that touched it, and reduce the blast radius so the next one is smaller (The Secret Lifecycle).
- Recovering from over-restriction is easy and should be made easy on purpose, because a governance system that is painful to correct is one that teams will bypass rather than fix.
- Recovering a lost declaration — an owner who left, a classification nobody remembers deciding — means re-deriving it from lineage: what does this column come from, and what was that classified as (Data Lineage)?
What can go wrong
- The policy exists and enforces nothing. The most common failure by a wide margin, and it is invisible until an audit or a breach.
- Enforcement exists in the warehouse only, and the lake underneath it is world-readable inside the company (The Data Lake).
- Classification propagation has a gap at one Python model, and every derived dataset below it is unclassified while looking governed.
- The retention job has been failing for months. Nobody alerted on it because nothing downstream depends on it succeeding — expiry is the one pipeline whose failure produces no missing data (Pipeline Observability).
- Governance is so slow that the fastest route to data is a colleague with a CSV, and the platform is now less governed than it was before the programme started.
- The service account that runs every transformation has read access to every dataset, so the platform's real access model is "whoever can modify a pipeline can read everything" (Data Access Control).
- "Governance is a compliance concern, so it is not engineering work." Every mechanism in this lesson is code, infrastructure or a scheduled job. The compliance function decides *what*; the platform decides whether the what is real.
- "We have a data governance policy" as an answer to "who can read this table". The policy is a declaration. The grant is the answer, and the two are frequently different.
- "Access control is the whole of governance." Access is one of five mechanisms and the only one people implement. Retention and deletion are where organisations are actually exposed, because unlike access they accumulate (Data Retention, Deletion Requests).
- "The warehouse enforces it, so we are covered." The warehouse enforces warehouse queries. Every data platform has at least one path to the same bytes that does not go through it (Object Storage as Data Infrastructure).
- This lesson is the frame for the module: classification declares, access enforces, minimization and retention bound how much there is to protect and for how long, masking reduces the value of what remains, and deletion is the obligation that tests whether any of it was real.
- The single measurement that predicts whether a governance programme works is the ratio of governed to ungoverned paths to the same data. Reduce the number of paths before improving controls on one of them.
- Auditability is a data engineering deliverable, not a security one. The access logs are a dataset with a schema, a grain, a freshness and a retention — and they are usually the worst-modelled dataset in the platform (Audit Logs for Privileged Actions).
Operating it
- Percentage of datasets in the warehouse that appear in the catalog with an owner and a classification — the coverage number that bounds every other governance metric.
- Count of datasets whose classification was inherited through lineage versus declared by a human, and the count where lineage could not be resolved. The third number is where the risk is (Impact Analysis).
- Access-log queries against classified-sensitive datasets, grouped by principal, with service accounts separated from humans. The interesting row is always a human account reading at machine frequency (Audit Logs for Privileged Actions).
- Age of the oldest object under each retention-governed prefix, compared with its stated retention. One query, and it is the only proof that expiry is running.
- At 10x datasets, manual declaration stops working and registration must be part of the publishing path rather than an afterwards.
- At 100x, classification must be inherited rather than assigned, and the quality of column-level lineage becomes the binding constraint on the entire governance programme (Column-Level Lineage).
- Consumer count scales the access-review problem faster than dataset count does: reviews are pairs, and pairs grow with the product of the two.
- Multiple regions or legal entities changes the problem in kind rather than degree — the same logical dataset now has different rules depending on where the row came from, which is a row-level policy question (Row and Column Security).
- Governance costs are mostly metadata and scan costs: cataloguing, classification scans over samples, and access logs that are themselves a large, long-retained dataset.
- Access logs are the sleeper. They are high-volume, they are retained longer than most business data because that is the point of them, and they are queried rarely — a textbook case for tiering rather than warehouse storage (Storage Lifecycle).
- The largest cost is human: review, approval, ownership. That is the cost that decides whether people route around the system, so automation of the common approval is the highest-leverage spend in the whole area.
- Every enforcement point you add is a place a legitimate query can be wrongly denied, and denial failures are debugged by the person least equipped to debug them. Fewer, better-placed enforcement points beat many partial ones.
- Tag-driven policy is more flexible than physically separating sensitive data, and physically separating it is more robust. Tags fail open when a tag is missing; separate storage fails closed. Highly sensitive data is the case where the rigid answer is right.
- Cataloguing everything makes the catalog authoritative and makes it noisy. A catalog of four thousand mostly-worthless staging models trains people to ignore it, which costs you the discovery value you were paying for (The Data Catalog).
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- ORG-SPECIFICClassification tiers, who counts as an owner, what an approval requires and which regulations apply are organisational and jurisdictional decisions. What generalises is the five-mechanism decomposition and the observation that programmes implement declaration and skip enforcement.
- GENERALThe structural point — that a copy inherits obligations but not enforcement, and that governance must therefore attach to metadata rather than to storage locations — holds for every platform regardless of stack or regulator.
- WAREHOUSE-SPECIFICWhether tag-based policies, row filters and column masks are native features or must be built from views differs sharply between warehouses, and a design that assumes native policy support will not port to an engine that only has grants on objects.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns how policy-as-code is reviewed, versioned and rolled out. A row filter deployed without review is a production change that can silently expose or silently hide data, and it deserves the same delivery discipline as application code.
- — Distributed Systems owns what it means for a revocation to propagate across regions and caches. A grant removed in one place and honoured everywhere is a consistency problem, and the window where it is not yet honoured is where incidents live.