Data Access Control
Least privilege applied to five surfaces — warehouse, lake, catalog, pipelines and secrets — where the weakest one is the effective policy and the pipeline is the most over-privileged principal you have.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Who can read this dataset, through every path that reaches its bytes, and how would you find out?
Two populations with different needs. Humans want the narrowest access that lets them work and want it today, because a slow grant is routed around. Workloads want a stable, narrow, auditable identity, and they get the opposite: one shared service account with read access to everything, because that is what makes pipelines stop failing (Human vs Workload Identity).
The unit of an access decision is (principal, dataset, action) — and in a governed platform, (principal, dataset, column-or-row, action). The grain most platforms actually implement is (group, schema, read), which is three levels coarser than the questions people ask of it, and that gap is where over-provisioning lives.
Create a few groups — analysts, engineers, admins — grant them on schemas in the warehouse, and give the orchestrator a service account with broad read so pipelines never fail on permissions. It is simple, it is comprehensible, and for a small team it is the correct amount of machinery.
The warehouse is one path to the bytes. The object store underneath it is another, and anyone with lake credentials reads the same Parquet files with no row filter, no column mask and no query log (Object Storage as Data Infrastructure).
- The warehouse is one path to the bytes. The object store underneath it is another, and anyone with lake credentials reads the same Parquet files with no row filter, no column mask and no query log (Object Storage as Data Infrastructure).
- The orchestrator's service account can read every source and write every target, because that is what an orchestrator does. The platform's effective access model is therefore "anyone who can merge a pipeline change can read everything", which is rarely the model anyone described (Orchestration).
- Access is granted per schema, and one table in that schema gains a personal column. The grant that was appropriate for the schema is now inappropriate for one table in it, and nothing re-evaluates it (Data Classification).
- A person moves teams. Their old grants persist because grants are additive and removal has no trigger, and after three moves they can read more than anyone intended (Least Privilege).
- The BI tool connects with a single technical account and does its own authorization internally. From the warehouse's perspective every dashboard is one principal, and the query log attributes everything to it (Audit Logs for Privileged Actions).
- A grant is removed and a cached extract, an open session and a materialised dashboard continue serving the data for hours or days afterwards (Sessions).
- Warehouse credentials end up in a notebook, a repository or an environment variable in a personal project, and are long-lived because rotating them breaks pipelines (Roles vs Static Keys).
What is actually happening
- A data platform has at least five access surfaces, governed by different systems that do not know about each other: the warehouse (grants, policies, row filters, column masks), the object store (cloud IAM on buckets and prefixes), the catalog (which may hold policy or merely metadata), the pipelines (the identities that read and write everything), and the secrets that let all of the above authenticate (Secrets Management).
- The effective access to a dataset is the union across all five, not the intersection, and the platform's real posture is set by the weakest one. This is why a beautifully governed warehouse over a permissive bucket is a common and completely ineffective design.
- Grants are additive and removals are manual, so access monotonically increases unless something forces re-evaluation. Time-bounded grants and periodic review are the only two mechanisms that reverse the drift, and only the first works without human effort (Short-Lived Credentials).
- Workload identity and human identity behave differently and should not share a model. A workload needs narrow, stable, machine-issued credentials scoped to the datasets one pipeline touches; a human needs broad-ish, short-lived, individually attributable access. Sharing a service account between the two destroys attribution entirely (Human vs Machine Identities).
- Role-based access maps principals to roles to datasets and works well until the number of distinct access shapes exceeds the number of roles anyone will maintain. Attribute-based policy — grant on classification, on ownership, on region — scales better and is harder to reason about (ABAC and Policy-Based Authorization, Role-Based Access Control).
- Every enforcement point that is bypassable is not an enforcement point. The design question is therefore how few paths reach the bytes, and whether the ones that remain evaluate policy (Where the Check Belongs).
Five surfaces, and the effective policy is the weakest
Ask "who can read fct_orders" and the natural move is to list the warehouse grants. That answer is correct about one surface and silent about four. The bytes are files in an object store with their own permissions. The catalog may or may not enforce anything. Several pipelines read the table with identities that are not people. And every one of those authenticates with a secret that is itself an access path.
The five rows below are the surfaces, what each one can express, and the specific way each one is usually wrong. Read the last column: the failures are not exotic. They are the default configuration of a platform assembled by competent people who governed the surface they were looking at.
The design consequence is to reduce the number of surfaces that humans can reach rather than to perfect the policy on each. A lake reachable only by engines has one fewer surface to govern, and the policy in the warehouse becomes load-bearing instead of advisory (Least Privilege in Infrastructure).
| Surface | Governed by | Granularity it can express | How it is usually wrong |
|---|---|---|---|
| Warehouse | Grants, roles, and native row and column policies | Database, schema, table, column, row | It is the only surface anyone configured, so it is treated as the whole answer while the files underneath are open. |
| Object store / lake | Cloud IAM on buckets and prefixes | Prefix or object — no notion of a row or a column | Broad read for a "data" group that includes everyone technical, bypassing every warehouse policy in one step (The IAM Model). |
| Catalog | Its own model, sometimes advisory only | Dataset metadata, and in some catalogs the policy itself | Assumed to enforce because it displays classifications. A catalog that only describes denies nothing (The Data Catalog). |
| Pipelines | Workload identity assumed by the orchestrator | Whatever the identity is scoped to, which is usually everything | One shared account with read on all sources and write on all targets, so pipeline-change access equals data access (Orchestration). |
| Secrets | A secrets manager, or environment variables and repositories | Whoever can read the secret has whatever it authenticates | Long-lived keys that nobody rotates because rotation breaks pipelines, stored where the orchestrator — and its users — can read them (Secrets Management). |
The pipeline is your widest principal
Draw the identities in a data platform and one of them reads every source and writes every target. It has to: an orchestrator that could not read a source could not run the pipeline that reads it. The result is that the most privileged principal in the company is a service account whose credentials live wherever the orchestrator can reach them, and whose actions are attributed to itself rather than to the person who caused them (Human vs Workload Identity).
This converts an access-control question into a code-review question. If merging a change to a pipeline definition means running arbitrary SQL as the account that can read everything, then the effective read policy of the platform is the merge policy of the pipeline repository, and that is rarely what anyone designed or reviewed (Privilege Separation).
The fix is unglamorous: one identity per pipeline, scoped to the datasets that pipeline names, issued short-lived, and derived from the pipeline's own declared inputs and outputs so that scoping is a by-product of the dependency graph rather than a separate manifest to maintain (Task Dependencies).
Choosing where the policy lives
Every access decision has to be evaluated somewhere, and the choice of where determines what happens when someone approaches the data by a path you did not anticipate. The options below are not a ladder — a large platform runs several at once — but they differ sharply in their bypass behaviour, and that is the property to select on (Where the Check Belongs).
The criterion that matters most is what happens to a path nobody tested. A policy in a warehouse view protects warehouse queries and nothing else. A policy in a catalog-backed engine layer protects everything that goes through that engine. Physical separation protects everything, at the cost of duplicating data and losing the ability to query across the boundary.
The option to reject firmly is the last one. Enforcement by convention — "analysts know to add WHERE region = ..." — is not a control, it is a naming convention with consequences, and it fails silently in the direction of exposure (Row and Column Security).
A dataset needs restricted access. Which enforcement point?
when Access is whole-table and the warehouse is the only path humans have to the data.
cost No column or row granularity, so a single sensitive column forces the whole table to the higher restriction and pushes people toward copies.
when The warehouse supports policy attached to tags, and the same table must serve differently-scoped readers.
cost Warehouse-specific and not portable; policies are code that can be wrong, and a wrong row filter changes query results rather than denying them (Row and Column Security).
when The engine has no native policy support, and you can prevent access to base tables entirely.
cost View sprawl, and the guarantee collapses the moment anyone is granted on a base table or reads the underlying files.
when Several engines read the same lake and you need one policy that all of them honour.
cost A component in the query path with its own availability, and any engine that can reach storage directly bypasses it entirely.
when The data is restricted or special-category and the cost of a mistake is unacceptable — the case where rigid beats flexible.
cost Duplicated storage, a separate pipeline, and no cross-boundary joins. Deliberate friction, which is the point (Workload Isolation).
when Never. Listed because it is what most platforms actually do before someone notices.
cost Unenforceable, unauditable, and it fails open: a forgotten predicate returns more data rather than an error.
How to build it
Most important first.
- Make the object store readable only by the engines. Humans and BI tools reach data through a system that evaluates policy; nobody has lake credentials as a matter of routine. This single change closes the largest hole in most platforms (Least Privilege in Infrastructure).
- Give every pipeline its own identity, scoped to the datasets it actually reads and writes. A shared orchestrator account is convenient precisely because it removes the constraint you are trying to impose (Anatomy of a Policy).
- Grant on classification and ownership rather than on object names, so a new table in a schema inherits the right policy instead of the schema's existing one (ABAC and Policy-Based Authorization).
- Make grants expire. A grant with an end date removes the need for a review process to catch it, and access that is still needed is re-requested — which is a much smaller amount of work than reviewing everything periodically (Short-Lived Credentials).
- Propagate the end user's identity through the BI tool where the tool supports it, so the warehouse sees a person and its policies and logs are meaningful. Where it does not, the BI tool becomes an enforcement point and must be governed as one (Where the Check Belongs).
- Use short-lived, issued credentials for everything and long-lived keys for nothing. The reason long-lived keys survive is that rotation breaks pipelines, so fix rotation rather than accepting the keys (Roles vs Static Keys, The Secret Lifecycle).
- Log at every surface, and unify the logs enough that "who read this dataset" is one query rather than four (Audit Trails).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- A warehouse grant guarantees that queries executed through that warehouse are subject to it. Nothing more — not the files it reads, not a replica, not a backup, not an extract already taken.
- Bucket IAM guarantees that requests through the object-store API are subject to it, at object or prefix granularity. It has no notion of rows or columns, so a lake path is all-or-nothing for any file it exposes (The IAM Model).
- Revocation guarantees no *new* authorization decisions succeed. Existing sessions, cached results and materialised extracts are outside it, and the window is bounded by whatever the caching layer decides (Caching Patterns).
- Nothing guarantees that a principal has not already copied the data. Access control governs reads, not what happens after one; that is what the copy inventory is for (PII in Pipelines).
- Least privilege is a property of a configuration at a point in time, not an invariant. Without expiry or review it degrades continuously and silently.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The reachability query: for each dataset classified above
internal, enumerate every principal that can reach its bytes through any surface — warehouse grants, bucket policies, pipeline identities and the BI service account — and compare against the intended list. The union is almost always larger than anyone expects. - It misses copies: a principal who cannot read the table but can read a mart derived from it, an extract in a bucket, or a dashboard that materialises it. Reachability must be computed over lineage, not over one dataset (Column-Level Lineage).
- It also misses transitive privilege — a principal who cannot read the data but can modify the pipeline that writes it, or assume the role that can. That is a graph problem and most platforms do not model it (Privilege Separation).
- Grant propagation is not instant across surfaces. Identity-provider group membership, warehouse role assignment and cloud IAM each converge on their own schedule, so a newly added person may have partial access for a period — and a removed one may retain partial access for the same reason.
- Revocation latency is the number that matters and the one nobody measures. Between removing a grant and the last cached result expiring, the data is still being served.
- Time-bounded grants convert this from a monitoring problem into a scheduling one, which is a much better shape: expiry happens on a clock instead of depending on someone noticing.
- A new table in an existing schema inherits the schema's grants, which is exactly the wrong default when the new table is more sensitive than its neighbours. Grant on classification and this stops being a hazard (Data Classification).
- A column added to an existing table is covered by whatever grant already exists on that table, so schema evolution is an access-control event even though nothing in the access system changed (Schema Evolution).
- Reorganisations change who should have access at a rate no manual process matches. Deriving grants from an identity provider's group membership makes the reorganisation the trigger, which is the only thing that keeps up.
- Over-provisioning is corrected by revoking, but revocation does not undo reads. Correct it and then determine what was accessed during the window using the access logs, which is the reason the logs exist (Audit Logs for Privileged Actions).
- Under-provisioning is corrected in minutes if the grant path is fast, and in two weeks if it is not — and the two-week version is what causes the extract-over-Slack workaround. Grant latency is a security control (The Self-Service Data Platform).
- A leaked credential is recovered by rotation, which is only fast if rotation was already automated. Practise it; the first rotation during an incident is not the time to discover which pipelines hard-coded the key (The Secret Lifecycle).
What can go wrong
- The warehouse is governed and the lake beneath it is company-readable, so every policy in the warehouse is advisory.
- One service account runs everything, so nothing can be attributed and the account cannot be scoped down without breaking an unknown set of pipelines.
- Grants accumulate across role changes and nobody has ever removed one. The access review, when it finally runs, is too large to act on and is rubber-stamped.
- The BI tool holds a technical account, so the warehouse log shows one principal reading everything and dashboard-level authorization is enforced by a tool nobody in the data team administers.
- A row filter or column mask is added and one path — a direct file read, a federated engine, a replication job — does not evaluate it, so the policy holds for the paths that were tested (Federated Query).
- Access requests are slow, so the fastest path to data is a colleague, and the platform's effective access model becomes social (Data Discovery).
- Break-glass access exists, is not time-bounded, is not alerted on, and quietly becomes normal access for the two people who know about it.
- "We use RBAC, so we have least privilege." Roles describe intended access; least privilege is a claim about actual access, which includes every surface, every pipeline identity and every historical grant nobody removed (Role-Based Access Control).
- "The service account is fine, it is not a person." It is the principal with the widest access on the platform and the credentials sit wherever the orchestrator can read them. It is the most attractive target you have (Secrets Are Not Configuration).
- "Access is controlled in the BI tool." Then it is controlled for people who use the BI tool. Anyone with a warehouse connection string is outside it (Where the Check Belongs).
- "We reviewed access last quarter." A review is a snapshot; grants accumulate continuously. Without expiry the review has to be repeated forever, and it will be skipped exactly when the platform is busiest.
- "Read-only access is low risk." Read is the whole risk for a data platform. Write access threatens correctness; read access is the exposure (Confidentiality, Integrity, Availability — Practically).
- Access is the mechanism people implement first and the only one they measure, which makes it a decent proxy for governance maturity and a poor substitute for it. Retention and deletion are where unmeasured exposure accumulates (Data Retention).
- Attribution is a governance requirement, not a security nicety: without per-human identity in the query log, no access question can ever be answered, and shared technical accounts destroy that at the source (Audit Logs for Privileged Actions).
- The pipeline identity is the governance gap nobody writes on the diagram. Scoping it per pipeline is the single highest-value access change most platforms can make, and it is usually deferred because it breaks things loudly (Orchestration).
Operating it
- Reachable-principal count per classified dataset, computed across all surfaces and tracked over time. A number that only goes up is a platform without revocation (Audit Trails).
- Grants with no expiry, and grants unused for a long window — the two lists that together produce almost every safe revocation (Least Privilege).
- Reads of sensitive datasets by human principals, separated from service accounts, with volume per principal. The interesting signal is a human reading at machine frequency, which is usually an export (Audit Logs for Privileged Actions).
- Break-glass usage, alerted in real time rather than reviewed monthly, because the point of break-glass is that it is exceptional.
- Credential age and rotation lag per pipeline identity, which is the metric that shows long-lived keys accumulating (The Secret Lifecycle).
- At 10x principals, group-based grants managed by hand stop working and access must be derived from identity-provider membership.
- At 10x datasets, object-name-based grants stop working and policy must be attached to classification and ownership tags (The Data Catalog).
- At 100x, the review problem dominates: reviewing (principal, dataset) pairs is quadratic in the two, and only expiry-by-default reduces it to something a human can act on.
- More engines is the underrated scaling axis. Each new query engine, federated connector or notebook runtime is a new path to the bytes with its own policy support, and the platform's posture drops to the weakest of them (Query Engines).
- Access control has almost no compute cost and a large human cost: request handling, approvals, reviews and the debugging of denials, all of which scale with principals multiplied by datasets.
- Audit logs are the storage driver, and they are high-volume, long-retained and rarely queried — the classic candidate for tiering rather than warehouse residence (Storage Lifecycle).
- Fine-grained policies add per-query work in the engine: a row filter is an extra predicate, and a masking policy is an extra expression per column. Relative and usually small, but it lands on every query rather than on some (Row and Column Security).
- The cost that dominates is the slow-grant tax. Every hour of grant latency is an hour of analyst time and a nudge toward an ungoverned copy, and it does not appear on any bill.
- Narrow, per-pipeline identities are correct and make pipelines fail on permissions during development. That friction is the mechanism working, and teams will ask for the shared account back.
- Attribute-based policy scales and is harder to reason about: answering "who can read this" becomes a policy evaluation rather than a lookup, and debugging a denial requires tooling that most platforms do not build (ABAC and Policy-Based Authorization).
- Short-lived credentials remove the standing risk of a leaked key and add a dependency on the issuing system in the ingest path. When that system is down, nothing runs.
- Closing the lake to humans is the highest-value change here and it removes an escape hatch that people use during incidents. Provide a break-glass path, time-bounded and alerted, or the door will be propped open permanently.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe five-surface decomposition and the union-not-intersection property hold for every platform, because they follow from the fact that analytical data is stored in one place and served through several.
- WAREHOUSE-SPECIFICWhether policies attach to tags, whether row filters and column masks are native, and whether the end user's identity can be propagated from a BI tool differ substantially between warehouses; a design assuming tag-based policy will not port to an engine with only object grants.
- CLOUD-SPECIFICObject-store permissions differ in model — prefix policies, bucket policies, per-object ACLs, and identity federation for workloads — so the same "lock the lake down" instruction has materially different implementations and failure modes across providers.
- ORG-SPECIFICWho approves access, how long a grant may last and whether break-glass exists are organisational decisions; the engineering supports any of them and the choice determines whether people use the governed path or route around it.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns credential issuance, rotation and the identity federation that makes short-lived workload credentials possible. Every argument for narrow pipeline identities depends on that machinery existing.
- — Distributed Systems owns why a revocation is not instantaneous across regions, caches and sessions, and how large that window can be.