ETL/ELTGENERALTOOL-SPECIFICSCALE-SPECIFIC

Raw, Staging, Curated: Layers by Purpose

Three jobs that need separating — preserve what arrived, make it usable, model it for consumers. The names vary by house; the purposes do not.

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

Why does the same data get written three times on its way from a connector to a dashboard, and what is each copy actually for?

Who needs this

Different consumers per layer, which is the whole point. Raw serves the pipeline itself and nobody else. Staging serves the modeller who needs conformed inputs. Curated serves analysts, dashboards, exports and models, and is the only layer that carries a contract anyone outside the data team should rely on (Data Contracts).

What one row is

The grain changes at each boundary and that change is the layer's justification. Raw holds the *source's* grain — one CDC change, one API item, one log line. Staging holds one row per business entity, reconstructed. Curated holds a declared analytical grain: one order, one order line, one customer-day. A layer that does not change the grain or the contract is a copy with a name (Grain: What Does One Row Represent?).

The obvious build

Write the transformation as one query from the connector's tables straight into the table the dashboard reads. It is one object to maintain, one place to look, and it removes two copies of the data. For a single small source this is not obviously wrong and is frequently the right starting point.

Why it breaks

The single query does deduplication, type casting, business filtering, joining and aggregation in one statement. When the number is wrong, there is no intermediate result to inspect, so debugging means commenting out pieces of a large query until it changes (Where Did This Number Come From?).

How it breaks with real data
  • The single query does deduplication, type casting, business filtering, joining and aggregation in one statement. When the number is wrong, there is no intermediate result to inspect, so debugging means commenting out pieces of a large query until it changes (Where Did This Number Come From?).
  • A second consumer needs the same cleaned data at a different grain. There is nothing to reuse, so the cleaning logic is copied — and the two copies diverge on the first edit (Two Dashboards, Two Numbers).
  • A source column is renamed. The query breaks in one place and the fix is applied there; six months later a different query, built from the same source, is found still using the old assumption.
  • The transformation is wrong for a month. Fixing it needs the source data as it was, and the only copy was the connector's tables, which mirror current state (Keeping Raw History: The Recovery Position and the Liability).
  • A deletion request arrives and no one can enumerate where a subject's data lives, because the data flows through a query rather than through named, catalogued datasets (Deletion Requests).
  • A test fails. It cannot say whether the problem is upstream data or downstream logic, because both live in the same object and there is no boundary at which to assert (Data Tests).
SourceIngestionRawTransformationValidationStorage ModelServingConsumerObservability

What is actually happening

  • Layers exist for reprocessing boundaries and contract boundaries, not for tidiness. A boundary is worth a layer when you would want to re-run everything after it without re-running everything before it, or when a group of consumers needs a stable shape that the layer above does not provide (Model Layering).
  • Raw preserves what arrived, unmodified, so every later stage is reconstructable. Its promise is fidelity to delivery, not correctness — it contains duplicates, source-shaped names, and whatever the connector delivered including its mistakes (The Raw Landing Zone).
  • Staging makes the data usable without deciding what it means: dedupe, cast types, rename to house conventions, handle deletes, apply nothing that a business person would recognise as a rule. Its promise is one row per entity with predictable types.
  • Curated applies meaning: business rules, joins across sources, the analytical grain, the metric definitions. Its promise is a documented contract — grain, owner, freshness, known gaps — that consumers may build on (Dataset Documentation).
  • The split matters most during an incident, because each boundary is a place to ask one question and get one answer. Is raw complete for the period? Is staging one row per entity? Is curated at the grain it claims? The first "no" is where the incident lives (Lineage Debugging).
  • The names are conventions and they vary — raw/staging/curated, landing/base/marts, bronze/silver/gold, source/intermediate/reporting. They describe the same three jobs. Arguing about the names is a substitute for arguing about the boundaries (Medallion: One Naming Convention Among Several).

Three jobs, and why they resist being done at once

The argument for layers is not aesthetic. Each layer is a place you can stop, look, assert something, and restart from — and a single query from connector to dashboard offers none of those. The moment a number is disputed, a layered platform has three places to check and a single-query platform has one large statement and a hypothesis.

The second argument is reuse. Deduplication and type conformance are needed by every consumer of a source; business rules are needed by some. Doing the first once and the second per consumer is the whole justification for the middle layer, and it is why "no business logic in staging" is a rule with teeth rather than a style preference.

The third argument is the one that matters most and shows up last. When a transformation is found to have been wrong for months, the fix requires an input that predates the mistake. That is what raw is, and it is the only layer whose absence cannot be remedied later.

Three layers, three promises, three ways to fail
  1. 1
    Raw

    Stores what the connector delivered, unmodified, partitioned by load time, append-only.

    guarantees Fidelity to delivery: whatever arrived is retained byte for byte and can be reprocessed. Nothing about completeness, duplicates or types.

    fails by Being configured to mirror current state instead of appending changes, which silently converts the history you are relying on into a snapshot (Snapshot and Stream: the Bootstrap Problem).

  2. 2
    Staging

    Deduplicates, casts types, renames to house conventions, represents deletes. No business rules.

    guarantees One row per business entity with predictable types, reusable by consumers who disagree about meaning.

    fails by Accumulating business rules until its contract is false, at which point every consumer inherits a filter nobody documented.

  3. 3
    Curated

    Applies business rules, joins across sources, sets the analytical grain, defines metrics.

    guarantees Its declared contract: grain, owner, freshness, known gaps. Nothing that is not declared.

    fails by Having no declared grain, so consumers infer one from the table name and aggregate against an assumption.

  4. 4
    Serving / mart

    Optional. A narrower, pre-aggregated copy for a specific heavy read pattern.

    guarantees Query cost and simplicity for one access pattern, at the price of flexibility for every other (Data Marts).

    fails by Drifting from its parent model after being patched directly during an incident and never reconciled.

Only the last stage is genuinely optional. The first three are jobs that must happen somewhere; layering is the decision to give each of them its own name, its own tests and its own place to restart from.

warehouse/
├── raw/                    ← what the connector delivered. append-only.
│   ├── raw_orders/         ← grain: one CDC change record
│   │   ├── _load_date=2026-03-13/
│   │   └── _load_date=2026-03-14/
│   └── raw_customers/
├── staging/                ← usable, not yet meaningful
│   ├── stg_orders          ← grain: one order, latest change, typed
│   └── stg_customers       ← grain: one customer
├── intermediate/           ← optional. shared joins, materialised once
│   └── int_orders_enriched
└── curated/                ← contract lives here. consumers read only this.
    ├── fct_orders          ← grain: one order. owner: revenue team.
    ├── dim_customer        ← grain: one customer, current
    └── revenue_daily       ← grain: one country-day (mart)

What one row means at each layer

GENERALThe grain progression from source-shaped records to a declared analytical grain holds regardless of naming or tooling; what varies is how many intermediate steps a platform uses, and whether the declared grain is written down anywhere a consumer can find it.

The layers are boundaries because the grain changes at them. That is the operational definition worth carrying: if two adjacent datasets hold the same grain, the same types and the same contract, one of them is redundant regardless of what it is called.

Read the breaksIf column as a list of real incidents. Every one of them is a query somebody wrote against a table that looked like it meant something else, and none of them produces an error — they produce a number, which is what makes them expensive.

The most common of these by a wide margin is the first: counting rows in a change-capture table and calling the result a business count. It is a reasonable thing to do to a table called raw_orders and it is wrong by a factor nobody can predict (What a CDC Event Contains).

One row, per layer
StageOne row isBreaks if
`raw_orders`One delivered change record — an insert, an update or a delete — possibly delivered more than once.You count rows and call them orders. Three updates to one order are three rows and one order, and a redelivery makes it four.
`stg_orders`One order, reconstructed as the latest change per order id, with types conformed and deletes represented."Latest" was chosen by arrival order rather than by the source's commit position, so an out-of-order update wins and the row is stale forever (CDC Ordering and Transaction Boundaries).
`int_orders_enriched`One order plus attributes joined from other staging models — still one order.A join to a dimension with duplicate keys fans out, so the count silently exceeds the number of orders and every measure downstream is multiplied.
`fct_orders`One order at the declared analytical grain, with business rules applied.Joined to an order-*line* fact without care, multiplying every order-level measure by its line count (Fact Tables).
`revenue_daily`One country-day, with revenue pre-aggregated.Someone joins it back to order-level data and re-aggregates, double counting in a way that looks like growth.

Four grain changes across five datasets. Each one is legitimate and each one is a place where a wrong assumption yields a plausible number rather than an error.

When a layer is not earning its keep

Layers are cheap to add and awkward to remove, so platforms accumulate them. The honest test for any layer is a pair of questions: does anything ever restart from here, and does any consumer depend on the shape this layer provides? A layer that answers no to both is a scheduled copy operation with a name.

The opposite failure is more damaging and harder to see: layers that exist in naming but not in behaviour. A curated model selecting straight from raw has skipped every assumption staging guarantees, and it will work perfectly until the day duplicates arrive. The convention held; the boundary never existed.

The failure table below is what layer problems look like from the outside. What they have in common is that none of them is reported as a layering problem — they are reported as a wrong number, a large bill, or a slow platform, and the layering is what the investigation eventually finds.

Layering problems, as they actually present
TriggerSymptomCauseResponse
A metric is several times its true value.Fact table row count exceeds the source's entity count.A curated model selecting directly from raw, at the change-record grain, skipping staging's deduplication.Enforce the boundary with grants and with the tool's dependency rules, not with a naming convention. Then add a grain assertion at the curated boundary so the next attempt fails the build.
A dashboard is fresher than the platform says it is.Numbers on one dashboard update between builds and disagree with the rest.A consumer pointed at a staging model to get better freshness, bypassing every business rule and test in curated.Find it in the query logs, then fix the cause: curated was not fresh enough for a real need. Restricting access without addressing the need just relocates the workaround.
Warehouse spend rises with no change in data volume.Build cost grows; scan volume per build is flat but the number of builds is not.An intermediate layer built nightly that nothing reads — created for a model that was since deleted.Audit reads per dataset from query logs and delete unread models. This is the one optimisation with no trade-off, and it needs lineage to do safely.
The same filter appears in eleven models.Business rule edits require eleven pull requests and one is always missed.A shared business rule with no home, because staging is forbidden to hold rules and no intermediate layer was created for it.Create the intermediate model. The "no business logic in staging" rule is right; the conclusion "therefore duplicate it everywhere" is not.
A schema change breaks six curated models at once.Six failed builds, one upstream rename.Curated models referencing source column names directly; the staging layer is not actually isolating anything.Route every curated model through staging so a rename breaks exactly one model. That isolation is the main thing the middle layer is for.
Two ways to decide whether a layer belongs
By reference architecture
The platform has the layers the diagram in the tool's documentation had. Every dataset is assigned to one, including datasets that pass through a layer unchanged because that is where the diagram said they go.
By boundary
A layer exists where you would want to restart processing, or where a group of consumers needs a contract the previous layer does not provide. Datasets that need neither skip the layer, and a layer nothing restarts from and nobody reads is deleted.

A pass-through layer costs a build and a copy on every run and provides no isolation, no reuse and no restart point — it is pure overhead that looks like architecture. The value of a layer is entirely in the boundary it creates, so a layer with no boundary has no value, whatever the reference diagram shows.

How to build it

Most important first.

  • Make raw immutable, append-only, partitioned by load time, and closed to consumers. Every property in that sentence is doing work: immutability makes it a reference, append-only makes it history, load partitioning makes reprocessing bounded, and the access restriction stops metrics being written at the source's grain (Data Access Control).
  • Keep business logic out of staging entirely. The test is simple: if a non-engineer would recognise the rule as a business decision, it belongs in curated. A WHERE status <> 'test' in staging is a business rule wearing a technical disguise.
  • Give every curated dataset a declared grain, an owner, a freshness expectation and a written list of what it does not include. An undocumented curated table is a staging table that consumers have been misled about (Data Ownership).
  • Add a layer only when you can name the boundary it creates. "We want an intermediate layer" is not a reason; "these five models share a join that costs more than storing it" is (Compute Waste).
  • Let the layer boundary be where tests live. Test raw for arrival completeness, staging for structural assumptions, curated for business assertions. Tests scattered without regard to layer end up asserting the same thing three times and the important thing nowhere.
  • Materialise by read-to-build ratio rather than by layer. There is no rule that staging must be views and curated must be tables; there is only the arithmetic of how often each is read (Where the Transformation Actually Runs).

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

  • Raw guarantees that what was delivered is retained and reprocessable. It guarantees nothing about completeness relative to the source, nothing about duplicates, and nothing about types (CDC Failure Modes and the Retention Deadline).
  • Staging guarantees structure: one row per entity, conformed types, deletes represented. It deliberately guarantees nothing about business meaning, which is what makes it reusable by consumers who disagree about meaning.
  • Curated guarantees its declared contract and nothing beyond it. A curated table with no declared grain guarantees nothing at all, whatever its name suggests.
  • No layer guarantees the layer above it was complete. Completeness is a property you measure by reconciling against the source, and every layer inherits the gaps of the one before (Reconciliation).
  • Publish atomicity is per dataset, so two curated tables built by the same run become visible at two different moments. A consumer joining them during that gap sees a state that never existed (Atomic Publish).

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this
  • Put a different class of check at each boundary. At raw: did the expected files or partitions arrive, and does the count match the source for a closed period. At staging: uniqueness on the entity key, not-null on the fields models assume, accepted values on enumerations. At curated: business assertions — revenue non-negative, every fact row joins to a dimension, the declared grain actually holds.
  • The grain assertion is the one most often missing and most often the cause. A single COUNT(*) versus COUNT(DISTINCT order_id) at the curated boundary catches every fan-out join in the platform (Duplicate Rows).
  • All of them miss meaning. Every layer can pass every check while the curated table reports gross revenue in a column named net_revenue, and no structural test will ever notice (Semantic Changes).
Freshness
  • Each layer adds its own build interval, and the consumer experiences the sum. Three layers built hourly are not hourly-fresh end to end; they are hourly plus the depth of the chain (The Transformation DAG).
  • The freshness gradient is a temptation: raw is always fresher than curated, so a consumer who needs freshness will find raw and use it, skipping every test and every business rule in between. The boundary has to be enforced with grants, not merely drawn on a diagram.
  • Layers can have different intervals deliberately. A curated model that drives an operational decision can build far more often than one feeding a monthly report, and running both on the same schedule wastes compute on one and freshness on the other (Cost vs Freshness).
When the schema or meaning changes
  • A new source column lands in raw automatically and stops there. Deciding whether it propagates to staging and curated is a deliberate act, and that gate is one of the main things layers buy you (Schema Evolution).
  • A renamed source column should break exactly one place — the staging model — and nothing downstream. If it breaks curated models directly, the layering is decorative rather than real.
  • A change in business meaning changes the curated layer and leaves raw and staging untouched, which is why a platform with layers can restate history and a platform without them cannot (Reprocessing vs Retrying).
  • Adding a layer is a breaking change for anyone reading the layer it displaces. Migrating consumers is the expensive part and it is always underestimated (Impact Analysis).
How to re-run this safely
  • Recovery walks down from the deepest intact layer: fix the model, rebuild from the layer above it, validate, publish. The layers are what make "rebuild from here" a meaningful instruction (Planning a Backfill).
  • A rebuild that starts from raw is always possible and rarely necessary. Starting from staging is cheaper and equally correct when the bug is a business rule, which most bugs are.
  • Rebuilding a middle layer without rebuilding its dependents leaves the platform internally inconsistent in a way that no single table's tests can detect. The dependency graph decides what must follow (Topological Execution).
  • Publishing a rebuild into the same tables consumers are reading is how a fix for March becomes an incident in September. Build aside, validate, swap (What Backfills Break).

What can go wrong

Failure modes
  • Layers that exist as naming conventions but not as boundaries — a curated model selecting directly from raw, bypassing the staging assumptions everything else relies on.
  • A staging layer that accumulated business rules, so the "no meaning here" contract is false and nobody knows which rules live where.
  • Consumers reading staging for freshness, and thereby reading data that no business test has been applied to.
  • The mitigation failing: access controls on raw that were applied at creation and never applied to the tables added later, so the boundary holds for the original five tables and not for the twentieth.
  • An intermediate layer created for one shared join that is now read by nothing, costing a build every night forever.
  • A layer boundary that exists in the warehouse and not in the catalog, so discovery surfaces raw tables to analysts as though they were products (Data Discovery).
Misreads
  • "More layers means better architecture." A layer is justified by a reprocessing boundary or a contract boundary. Platforms with seven layers usually have three that exist because a reference diagram had them.
  • "The layer names are the architecture." The names are a convention. Two platforms using identical names can have completely different boundaries, and two using different names can be structurally identical.
  • "Staging is where we clean the data." Staging is where you make it *usable*. Cleaning implies a judgement about what is correct, and that judgement is a business rule belonging in curated.
  • "Raw is just a backup." A backup is restored after a disaster. Raw is read on a normal Tuesday to fix a normal bug, which makes it a working dataset with a retention argument of its own.
  • "Consumers can read any layer as long as they know what they are doing." They will not know, because the grain of raw is not written on the table. Access restriction is the control; a naming convention is a hope.
Privacy, retention and access
  • Raw is the highest-risk layer in the platform: complete, source-shaped, unfiltered, and inside an analytical system. Its access should be the narrowest, which is the opposite of how most platforms configure it.
  • Classification applied at the raw boundary can propagate to every derived layer automatically; applied later it has to be re-derived per model by hand and will be incomplete (Data Classification).
  • Deletion has to reach every layer, and the layers are what make that enumerable at all. A platform with named, catalogued layers can answer "where does this subject appear"; one with a single query cannot.
  • Retention should be argued per layer with different reasoning: raw as a recovery window, curated as a consumer requirement. Applying one policy to both is how platforms keep everything forever or lose the ability to restate (Data Retention).

Operating it

How you see it in production
  • Row counts per layer per period on one chart. A drop between two adjacent layers localises a fault to one transformation instead of to a platform (Pipeline Metrics).
  • Freshness per layer, published separately, so the gap consumers actually experience is visible rather than inferred (Freshness Monitoring).
  • A query-log audit of who reads which layer. Consumers reading raw or staging is the single most useful governance signal a platform can collect, and it is available from logs nobody looks at (The Data Catalog).
  • Build cost per layer, so an intermediate layer that costs more than it saves can be identified rather than defended (Cost Attribution).
What changes at 10x and 100x
  • At 10x volume, full rebuilds of the lower layers stop fitting the schedule first, because they carry the most rows. Incrementality arrives from the bottom of the stack upward (Full Refresh vs Incremental).
  • At 100x, raw retention becomes a tiered decision — recent hot, older archived, oldest aggregated — and the tiering has to be designed so that a reprocess of an old period is slow rather than impossible.
  • At high model count, the middle layer is where sprawl happens: intermediate models multiply because each is individually justified. Periodic deletion of unread models is maintenance, not cleanup (Data Platform Anti-Patterns).
  • At high consumer count, the curated contract stops being documentation and becomes an interface with versioning problems of its own (Backward Compatibility).
What drives cost here
  • Each layer is a stored copy plus a build. Three layers means roughly three times the storage of one, which is usually the cheapest line in the platform and the one most often used to argue against layering (Storage Lifecycle).
  • The build cost is the real number, and it is dominated by layers that rebuild history rather than changes. A three-layer platform doing full refreshes pays for its layering three times over (Incremental Processing).
  • Layers *reduce* cost when they materialise shared work: a join used by ten models built once beats the same join computed ten times.
  • A layer that no consumer reads and no reprocess starts from is pure cost. Deleting such layers is the rare optimisation with no downside, and it requires knowing who reads what (Data Lineage).
What this approach costs
  • Layers buy debuggability, reuse and bounded reprocessing; they cost storage, build time, and a longer path from a source change to a dashboard change — which analysts experience as the data team being slow.
  • A strict "no business logic in staging" rule buys reusability and costs convenience: the analyst who needs test accounts excluded everywhere now has to have that rule applied in each curated model, or accept an extra layer for it.
  • Restricting raw to the pipeline buys correct metrics and costs the freshest data being unavailable to the people who most want it. That tension is permanent and is best managed by making curated fresh enough rather than by relaxing the boundary.

Layer boundary lab

Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.

Layer boundary lab
Place each piece of logic. Both directions are mistakes: too early destroys something you cannot get back, too late means it is done more than once and differently.
Placed
2 of 8
In the right layer
1
Too early
1
Too late
0
Rename `cust_id_num` to `customer_id`right layer
Renaming and typing is the entire job of the staging layer. Doing it once here means no downstream model has to know the source's vocabulary.
Keep the latest row per order id
Group events into sessions with a 30-minute gap
Subtract refunds from revenuetoo early
Netting refunds in staging destroys the ability to report gross revenue at all, and the source no longer reconciles.
Convert amounts to a reporting currency
Exclude internal test accounts
Format a number as a percentage with one decimal
Hash the email address
ORG-SPECIFICWhere a boundary sits is partly a team decision — a platform with one analytics team can merge staging and intermediate without harm, and a platform with eight cannot. The asymmetry does transfer: too early loses information permanently, too late loses agreement.

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

  • GENERALThe three purposes — preserve what arrived, make it structurally usable, apply meaning for consumers — exist in every analytical platform including ones with no formal layering, where they are simply performed by the same query and are correspondingly hard to debug.
  • TOOL-SPECIFICWhether layer boundaries are enforceable depends on the transformation tool: one that models dependencies as named references can forbid a curated model selecting from raw, while a platform of scheduled SQL statements can express the convention and enforce nothing.
  • SCALE-SPECIFICBelow a handful of sources and a few dozen models, two layers are usually enough and a third is overhead; the intermediate layer earns its place when several models share expensive joins or when the number of consumers makes a stable contract necessary.

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

DSAdag
Domains that do not exist yet
  • DevOps / Production Engineering owns the environment story that layering interacts with: whether a development build writes to a separate schema, and how a model is promoted from development to production without a consumer noticing.
  • Distributed Systems owns why the raw layer contains duplicates in the first place, and why designing them out of the delivery path is usually more expensive than deduplicating in staging.