The Lakehouse
Object storage for the bytes, a table metadata layer for the transactions, and independent query engines on top — a combination rather than a product.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Can files on object storage behave like tables, and what exactly is added to make that true?
Consumers who want warehouse behaviour — atomic publish, consistent reads, deletes, schema evolution — without giving up the ability to point a different engine at the same bytes tomorrow. In practice that means transformation jobs, SQL analysts and ML training pipelines all reading one copy instead of three (Who Actually Consumes This Data).
One row of a declared table, where the table is defined as a specific set of data files listed by a metadata snapshot rather than as a directory. That definition is the whole lesson: the grain is the same as a warehouse's, but membership in the table is explicit rather than implied by a prefix (Open Table Formats).
Keep files in the lake and point a query engine at the directory. It works immediately, costs nothing to set up, and for read-only historical data that is only ever appended by one writer it is entirely adequate — which is why so many platforms sit here for years.
A daily job rewrites a partition to correct a bug. Readers running during the rewrite see some old files and some new ones, and produce a number that never existed as a consistent state (Atomic Publish).
- A daily job rewrites a partition to correct a bug. Readers running during the rewrite see some old files and some new ones, and produce a number that never existed as a consistent state (Atomic Publish).
- Two jobs write to the same table at once. Neither fails; both succeed; the directory now contains the union of two writers' output and the row count is wrong in a way that looks like a business event.
- A GDPR deletion needs one customer removed. There is no
DELETE— the only mechanism is find the affected files, rewrite them without those rows, and swap, which is exactly the operation with no atomicity (Deletion Requests). - A finance dispute needs the table as it was three weeks ago. The directory holds only the current files, and the ones it replaced are gone.
- A producer adds a column. Old files do not have it, new files do, and the engine's behaviour on the union depends on whether it infers schema from the first file, the last file, or all of them (Schema Evolution).
- Query planning slows down as the table grows, because planning means listing a very large number of keys to find out what the table even contains (Object Storage as Data Infrastructure).
What is actually happening
- A lakehouse is three independent layers, and its defining property is that they are replaceable independently. Storage: objects in a bucket, usually columnar files (Parquet). Table metadata: a format that records which files constitute the table at each snapshot (Open Table Formats). Compute: one or more engines that read the metadata, then the files (Query Engines).
- The metadata layer is where every warehouse-like guarantee comes from. Because "the table" is a list rather than a prefix, adding, replacing or removing files is a change to a list — and a change to a list can be made atomically even though a change to many objects cannot.
- That single move buys atomic publish, snapshot isolation for readers, time travel, row-level deletes and safe concurrent writers, all on storage that offers none of them (Transactions and ACID).
- Statistics in the metadata — per-file and per-column min/max, null counts, row counts — let an engine skip files without opening them. Planning becomes a metadata read rather than a directory listing, which is why the pattern scales to tables a listing could not enumerate (Predicate Pushdown).
- What it deliberately does *not* add is an integrated optimiser, workload management, or a single governance surface. Those live in whichever engine you point at the table, and they differ per engine — which is the price of the openness (Lake vs Warehouse vs Lakehouse).
- The word describes a combination, not a category of product. A team that stores Parquet in a bucket, manages it with a table format and queries it with two engines has a lakehouse whether or not they bought anything (Data Architecture Patterns).
Three layers, replaceable independently
The value of the lakehouse framing is not that it is new but that it is decomposed. A warehouse fuses storage, table semantics and compute into one product; here they are three things with interfaces between them, and each can be swapped without the others noticing.
That decomposition is what makes the trade-offs legible. The bytes are yours, in an open format, in your bucket. The table semantics come from a metadata layer that is also just files in that bucket. The engine is the only part that is a running system, and it is the part you can replace on a Tuesday.
It is also what makes the responsibilities legible. Nobody is running compaction for you. Nobody is arbitrating between the ML job and the finance query. The layers give you guarantees about the *data*; they give you nothing about operations, and that work does not disappear because the architecture is open (Data Platform Engineering).
- The engines never talk to each other. Their only shared state is the catalog pointer, which is why two of them can read while a third writes (Compare-and-Swap: The Primitive Everything Is Built On).
- Statistics live in the manifests, so pruning benefits every engine equally instead of being an engine feature (Predicate Pushdown).
- The catalog is the single point of coordination and therefore the single point of failure for correctness — two catalogs over one table is the classic way to corrupt it.
What the metadata layer actually changes
It is easy to describe the lakehouse in terms of features gained. It is more useful to describe it in terms of one thing changed: the definition of "the table". In a plain lake, the table is whatever files happen to be under a prefix right now. In a lakehouse, the table is the file list a snapshot names.
Every difference follows from that. If membership is a list, then changing membership is one write, so it can be atomic. If old lists are kept, then old versions are queryable, so time travel exists. If the list carries per-file statistics, then planning can skip files without listing or opening them. If commits go through a compare-and-swap on a pointer, then two writers cannot both win.
The comparison below is the same bytes in both columns. The bucket is identical, the Parquet files are identical, the query is identical. What changed is who decides which of those files are the table, and that turns out to be the difference between a directory and a database.
What does this table need that a directory of Parquet files does not give it?
when Append-only, one writer, never rewritten, read by one engine, and no deletion obligation.
cost None, and you avoid compaction jobs, snapshot expiry and catalog operations. Revisit the moment a second writer or a delete appears.
when Consumers query the table while a job rewrites part of it, and a half-published state would be read as real.
cost A catalog and a commit protocol in the write path. Buys the ability to republish without a maintenance window.
when A streaming writer and a batch backfill both target the table, or several partitions are written in parallel by different jobs.
cost Commit conflicts and retries that the writers must handle. Buys correctness that directory writes cannot provide at all (Optimistic Concurrency Control).
when Deletion requests, late corrections, or a merge from a change stream (Change Data Capture).
cost Delete files and the compaction that eventually applies them; read amplification until it runs. Buys an operation that is otherwise a full-partition rewrite.
when Reports must be reproducible and a bad publish must be revertible in minutes.
cost Snapshot retention, which is storage. Buys the fastest rollback in the platform (Rolling Back Data).
when Planning time is dominated by listing keys rather than by reading data.
cost Metadata reads per query and metadata maintenance. Buys planning that scales with matching files rather than with total files.
The job deletes the eleven files under `dt=2026-08-24/` and writes fourteen new ones. Between the first delete and the last write, readers see a partition with anywhere from zero to fourteen files. A dashboard refreshing in that window reports a number that corresponds to no state the data was ever in, and there is no record afterwards that it happened.
The job writes fourteen new files under a new path, then commits a snapshot that removes the eleven old entries and adds the fourteen new ones. Readers holding the previous snapshot continue reading the old files until they finish. Readers arriving after the commit see the new set. The old snapshot remains queryable until expiry, so the previous number is reproducible during the dispute that follows.
Object storage can make a single small write atomic and cannot make eleven deletes and fourteen writes atomic. Moving the definition of the table into one small object converts an operation that has no atomicity into one that does — which is the entire mechanism, and the reason this works on storage that was never designed for tables (Open Table Formats).
Iceberg, Delta Lake and Hudi are the three widely deployed table formats and their capabilities have converged considerably, but their commit mechanics, delete representations and catalog requirements still differ, and engine support for each varies by engine version. Treat "engine X supports format Y" as something to verify against current documentation for the specific version you run, especially for delete files and schema evolution.
How to build it
Most important first.
- Adopt the table format at the boundary consumers read, not everywhere. Raw landing stays plain files — it is append-only, single-writer and never queried directly, so the metadata layer buys it nothing (The Raw Landing Zone).
- Pick one catalog and make it the only way tables are resolved. The commit protocol depends on the catalog; two catalogs over one table is two writers who cannot see each other (The Data Catalog).
- Schedule compaction and metadata expiry from day one. Both snapshots and small files accumulate, and the table gets slower in a way that looks like data growth but is not (File Compaction, File Size and the Small-Files Problem).
- Write partitioning that the format can evolve. The point of hidden or evolvable partitioning is that a bad partition choice stops being permanent, which it emphatically is with directory layouts (The Partitioning Decision).
- Keep deletes explicit and bounded. Row-level deletes are supported, but a delete that touches most files is a full rewrite wearing a different name (Upserts and Merges).
- Decide the snapshot retention window as a recovery decision. Time travel is your fastest rollback and it is only as long as the window you kept (Rolling Back Data).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Atomic commit of a set of file changes: readers see the snapshot before or the snapshot after, never a mixture. This is the guarantee the whole layer exists to provide (Atomic Publish).
- Snapshot isolation for readers — a long query reads one consistent version of the table even while writers commit (Isolation Levels).
- Serializable or optimistic-concurrent writes depending on configuration: concurrent writers detect conflicts and one retries, rather than both silently succeeding (Optimistic Concurrency Control).
- Schema evolution with stable column identity, so a rename is a metadata operation rather than a rewrite — for the formats that assign column ids (Schema Evolution).
- No guarantee about anything outside the table. A file written into the table's directory but not committed to a snapshot is invisible; a file deleted from the bucket out of band corrupts the table silently.
- No guarantee of cross-table transactions. Publishing two tables consistently is two commits, and something can read between them (Distributed Transactions).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check specific to this layer is metadata-versus-storage reconciliation: every file referenced by the current snapshot exists, and every file under the table's prefix is either referenced by a live snapshot or expired. Divergence in either direction is a corruption you want to find before a query does.
- It misses everything about the contents of those files. A structurally perfect table whose manifests are all consistent can still be missing a third of the source (Reconciliation).
- It also misses out-of-band writes that happened and were then cleaned up — the table can be silently wrong for a window and structurally fine afterwards, which is why write access to the bucket should be narrower than read access (Data Access Control).
- Commit granularity sets the floor on freshness: data is visible when a snapshot including it commits, not when the file lands. Streaming writes therefore trade commit frequency against metadata volume.
- Frequent commits give fresher data and more snapshots, more manifests and more small files — so the cost of freshness here is paid in planning time and background maintenance rather than in ingest (Cost vs Freshness).
- Readers can deliberately pin an older snapshot, which makes reproducibility a first-class option: two runs of the same report against the same snapshot id give the same answer, which is not otherwise true of a table being written to.
- Column add, drop, rename and reorder become metadata operations for formats that track column ids, so history does not need rewriting. That is a genuine improvement over directory-plus-Parquet, where a rename is either a rewrite or a positional accident (Backward Compatibility).
- Partition evolution means an early partitioning mistake can be corrected forward without rewriting history, which changes partitioning from an irreversible decision into a revisable one (Partitioning).
- None of that touches meaning. A column whose definition changed evolves perfectly and reports the wrong thing, exactly as it would anywhere else (Semantic Changes).
- Rolling back a bad publish is a metadata operation: point the table at the previous snapshot. This is the fastest rollback available anywhere in a data platform and it is the main operational reason to adopt the layer (Rolling Back Data).
- Backfills replace files inside a commit, so a corrected partition becomes visible atomically and the previous version remains queryable until snapshot expiry (Planning a Backfill).
- The recovery window is the snapshot retention window. Expire aggressively for storage and you have shortened your rollback horizon — that trade should be made deliberately, not by accepting a default.
What can go wrong
- Two writers using different catalogs for the same table, each committing successfully against its own view, silently diverging.
- A file deleted directly from the bucket — by a lifecycle rule or a well-meaning cleanup — while a live snapshot still references it, producing read errors that look like engine bugs (Storage Lifecycle).
- Snapshot and manifest accumulation from frequent commits, degrading planning time until someone runs expiry.
- An engine whose support for the format lags: it reads the table but ignores row-level deletes, so deleted rows reappear in that engine's results only.
- The mitigation failing: compaction itself rewriting files while a long-running reader holds an old snapshot, which is safe only because expiry has not yet removed those files — set expiry shorter than your longest query and you have built a race (Partial Failure).
- "A lakehouse is a product you buy." It is a combination of three layers. Several vendors sell an assembled version, which is a legitimate thing to buy, but the architecture exists independently of any of them.
- "Lakehouse means we do not need a warehouse." It closes the gap on transactions and performance and does not close it on integrated governance, workload isolation and optimiser maturity. Many platforms run both on purpose (Lake vs Warehouse vs Lakehouse).
- "Time travel is a backup." It is bounded by snapshot retention and it lives in the same storage as the data. A bucket-level disaster takes both (Backup Strategy).
- "Adding a table format fixed our small-file problem." It made the problem visible and gave you a safe way to compact. Compaction is still a job somebody has to schedule and pay for (File Compaction).
- "Any engine can read it, so we are portable." Format support varies by engine and by version, particularly for deletes and evolved schemas. Portability is a property to verify per engine, not to assume from a format name.
Operating it
- Snapshot count and manifest count per table over time. Rising planning latency almost always traces to one of these before it traces to data volume.
- Average data file size per partition, which is the direct signal for whether compaction is keeping up (File Size and the Small-Files Problem).
- Commit conflict and retry rate per table — the honest measure of whether your concurrent-writer story works (Optimistic Concurrency Control).
- Files referenced by live snapshots versus files present under the prefix, as a scheduled audit rather than an incident-time query.
- At 10x table size, metadata pruning is what keeps planning constant while listing would have grown linearly. This is the property that makes the layer worth its complexity (Partition Pruning).
- At 100x, manifest layout and compaction cadence become the operational problem, and the table format's own maintenance jobs become a scheduled workload with their own reliability requirements.
- Engine count scales the compatibility problem rather than the data problem: every additional engine is another implementation of the format whose support for deletes, evolution and statistics you have to verify (Federated Query).
- Metadata operations add read requests per query, which is a small cost that becomes visible on tables with very many snapshots or manifests (Object Storage as Data Infrastructure).
- Retained snapshots hold files that would otherwise be deleted, so recovery horizon is directly a storage bill (Storage Lifecycle).
- Compaction and expiry are background compute that never appears in any query's cost attribution and is therefore consistently under-budgeted (Compute Waste).
- Against that, pruning from file statistics reduces bytes scanned by every reader, and that saving compounds across engines because it lives in the table rather than in one engine (Predicate Pushdown).
- You gain transactions, time travel and engine independence, and you take on the maintenance of a table format — compaction, expiry, catalog operations — that a managed warehouse would have done invisibly (The Data Warehouse).
- Openness is real and it is not free. One copy readable by several engines means the governance and performance behaviour differ per engine, so the guarantees your analysts get depend on which tool they opened.
- Deletes are supported and remain expensive. Positional or equality deletes defer the rewrite; the rewrite still happens, and a delete-heavy workload on a lakehouse table is a workload fighting its storage model (Upserts and Merges).
Lakehouse layers
Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.
| Staging | |
|---|---|
| Promises | One model per source table, renamed, typed and deduplicated — and no business logic whatsoever. A join here is the first sign the layer has stopped meaning anything. |
| Models here | stg_orders, stg_customers, stg_events |
| Relative weight | 206.0Msim rows · 39.2%sim of the project |
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe three-layer decomposition and the metadata-pointer commit are common to every table format in this category. What differs is the file-level detail of manifests and the mechanics of delete files, not the fact that a commit is a pointer swap.
- FORMAT-SPECIFICIceberg, Delta Lake and Hudi differ in how they record deletes, how partitioning is expressed, and how the current-snapshot pointer is made atomic — Iceberg leans on a catalog's compare-and-swap, Delta on an ordered log of commit files. Those differences decide which catalogs and engines each can safely use.
- ENGINE-SPECIFICSupport for the same format varies by engine and version: an engine that reads data files but not delete files will return rows that another engine correctly hides, so the same table can give two answers depending on which tool you opened.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns what makes a compare-and-swap on a catalog entry a genuine linearisation point, and what happens to a commit protocol when the catalog itself is partitioned from a writer.
- — DevOps / Production Engineering owns running the table maintenance jobs — compaction, snapshot expiry, orphan-file cleanup — as scheduled production workloads with their own alerting.