ETL & ELT

Where transformation runs and what that decides. Not a fashion — a question about where compute lives, how much raw history you keep, and what you can reprocess after you find a bug.

ETL: Transform Before the Data Lands
▶ lab

Extract, transform, load. The destination only ever sees rows that already conform — a real guarantee for its readers, bought with the original that nobody kept.

Q · If the destination is perfectly capable of storing the raw rows, why would anyone transform the data before it gets there?
ELT: Load First, Transform Where the Data Lives
▶ lab

Extract, load, transform. The destination holds the raw copy and does the work — which turns most transformation bugs into a re-run and hands the destination every obligation the raw data carries.

Q · What does a pipeline gain by loading data it has not yet cleaned, and what does the destination inherit the moment it does?
ETL vs ELT: Choosing by Constraint, Not by Fashion
▶ lab

Two orderings, six criteria. Where compute is available, how large raw is, whether you need history, what security forbids, how complex the logic is, and what the destination can actually do.

Q · For this specific source, feeding this specific destination, which ordering does the constraint actually permit — and which one is a preference dressed as a principle?
Where the Transformation Actually Runs
▶ lab

In the source, in a dedicated cluster, in the warehouse, in the query at read time, or in the BI tool. Each placement moves cost, freshness, testability and governance somewhere different.

Q · A number needs filtering, joining and aggregating before a human sees it — so which machine should do that work, and what does putting it there decide?
Raw, Staging, Curated: Layers by Purpose
▶ lab

Three jobs that need separating — preserve what arrived, make it usable, model it for consumers. The names vary by house; the purposes do not.

Q · Why does the same data get written three times on its way from a connector to a dashboard, and what is each copy actually for?
Medallion: One Naming Convention Among Several
▶ lab

Bronze, silver, gold is a widely used set of names for raw, staging and curated. It is a convention, not a requirement, and it is not an architecture.

Q · What does calling a layer "bronze" tell a consumer that calling it "raw" does not — and what does the medal metaphor quietly imply that is false?
Keeping Raw History: The Recovery Position and the Liability
▶ lab

An immutable copy of what arrived is what makes every downstream mistake fixable. It is also the most sensitive dataset the platform holds. Both are true and neither cancels the other.

Q · What can you still fix a year from now, and what have you promised to be able to delete?