StorageGENERALFORMAT-SPECIFICENGINE-SPECIFIC

Open Table Formats

How Iceberg, Delta and Hudi turn immutable objects into a transactional table: a manifest of which files are the table now, and a commit that is one pointer swap.

What actually happens How to build it Can I trust it?▶ Run the lab

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

How can a set of immutable files on storage with no rename and no multi-object atomicity behave like a table with transactions?

Who needs this

Every engine reading the table, and every writer publishing to it. What they need is agreement — a single, unambiguous answer to "which files are this table right now" that does not depend on when they listed the directory (The Lakehouse).

What one row is

Two grains stack here, and confusing them is the most common source of confusion about the format. The data grain is a row inside a Parquet file. The metadata grain is one entry per data file in a manifest, carrying that file's partition values, row count and per-column statistics (Grain: What Does One Row Represent?).

The obvious build

Define the table as "every file under this prefix". Listing is the read path, appending is the write path, and nothing else is needed. For a single-writer append-only archive this is correct and adding machinery to it would be waste.

Why it breaks

A rewrite has no isolation. Replacing a partition means deleting and writing several objects, and any reader in between sees a set of files that was never a valid state of the table (Atomic Publish).

How it breaks with real data

A rewrite has no isolation. Replacing a partition means deleting and writing several objects, and any reader in between sees a set of files that was never a valid state of the table (Atomic Publish).
Two concurrent writers cannot detect each other. Both list, both write, both succeed, and the resulting file set reflects neither writer's intent.
Planning cost grows with total files rather than with matching files, because the only way to know what is in the table is to list it (Object Storage as Data Infrastructure).
A rename is not a rename. On object storage it is a copy followed by a delete, so any scheme that publishes by renaming a staging directory is publishing non-atomically and does not know it.
A column rename is either a rewrite of every file or a positional accident, because the table's schema is inferred from files rather than declared (Schema Evolution).
There is no previous version. Once the old files are deleted, the number that was on the dashboard yesterday cannot be reproduced (Where Did This Number Come From?).

Source→Ingestion→Raw→Transformation→Validation→Storage Model→Serving→Consumer→Observability

What is actually happening

The core idea is one indirection. Instead of "the table is the prefix", the format says the table is the file list named by the current snapshot, and the current snapshot is named by a single small pointer.
A snapshot names a set of manifests; each manifest lists data files with their partition values, record counts and per-column min/max and null counts. A file the statistics prove cannot match a predicate is skipped without ever being opened (Predicate Pushdown).
A commit is: write the new data files (nobody can see them — they are in no snapshot), write new manifests describing the resulting file set, write a new snapshot object, then atomically move the pointer from the old snapshot to the new one. That pointer move is the only operation that must be atomic, it is a single-object operation, and it is the one thing object storage does give you — so everything transactional about the table reduces to it (Compare-and-Swap: The Primitive Everything Is Built On, Immutability as a Concurrency Strategy).
Concurrency is optimistic. A writer reads the current snapshot id, does its work, and commits only if the pointer still holds the id it started from. If another writer got there first, the commit fails and the writer re-reads and retries (Optimistic Concurrency Control).
Time travel is free once you have this: old snapshots are still valid file lists, so reading "as of" a snapshot id or a timestamp is reading an older list. It costs storage, not machinery (Validating a Backfill Before You Publish).
Updates and deletes are expressed as either rewritten data files (copy-on-write) or small side files marking removed rows (merge-on-read), and the choice is a read-cost versus write-cost trade rather than a feature difference (Upserts and Merges).

The metadata tree, and what a commit changes in it

SIMPLIFIEDThe tree is drawn Iceberg-shaped for concreteness; Delta expresses the same idea as an ordered sequence of JSON commit files plus periodic checkpoints, where atomicity comes from the ordering of commit filenames rather than from a catalog compare-and-swap. The indirection is identical, the file mechanics are not.

The structure below is the whole mechanism drawn out. Read it bottom-up: data files hold rows, manifests list data files with statistics, a manifest list groups manifests into a snapshot, and one pointer says which snapshot is current.

Now read the commit. The writer produced two new data files and logically removed one. It wrote a new manifest, a new manifest list and a new snapshot — none of which any reader can see, because none of them is referenced by the pointer. The last step changes one small object, and at that instant every new reader sees the new table and every existing reader keeps reading the old one.

Everything else people associate with these formats falls out of this shape. Old snapshots are still valid lists, so time travel exists. Manifests carry statistics, so pruning exists. The pointer swap is conditional on the previous value, so concurrent writers conflict instead of clobbering. There is no additional machinery — this is it.

Steps 1 to 4 can fail at any point and leave the table untouched; the debris is orphan files, not corruption.
Step 5 is conditional. If another writer moved the pointer to snap-0003 first, this commit fails and retries from the new base (Compare-and-Swap: The Primitive Everything Is Built On).
manifest-a is reused unchanged — a commit rewrites the metadata it touches, not the whole inventory, which is why commit cost tracks the change and not the table.
The amount:[min, max] ranges are what let an engine skip a file for WHERE amount > 9000 without opening it (Predicate Pushdown).

catalog: warehouse.sales.fct_orders  ->  snap-0003          <- the only thing that must be atomic

  snap-0002  (still valid, still queryable)
    manifest-list-0002
      manifest-a  -> data/dt=2026-08-24/part-0000.parquet   rows=812k  amount:[0.00, 4199.00]
                     data/dt=2026-08-24/part-0001.parquet   rows=790k  amount:[0.00, 8650.00]
      manifest-b  -> data/dt=2026-08-25/part-0000.parquet   rows=804k  amount:[0.00, 5120.00]

  snap-0003  (current)
    manifest-list-0003
      manifest-a  -> data/dt=2026-08-24/part-0000.parquet   rows=812k     unchanged, reused
      manifest-c  -> data/dt=2026-08-25/part-0002.parquet   rows=511k     new
                     data/dt=2026-08-25/part-0003.parquet   rows=298k     new
                     (part-0001.parquet from dt=2026-08-24 no longer listed)

commit sequence:
  1. write new data files            <- invisible; referenced by nothing
  2. write manifest-c                <- invisible
  3. write manifest-list-0003        <- invisible
  4. write snap-0003                 <- invisible
  5. CAS catalog pointer 0002 -> 0003   <- the table changes here, atomically

What the indirection buys, in SQL you can actually write

The operations below are not new SQL — they are ordinary statements that were impossible over a directory of files, and each one is a direct consequence of the snapshot indirection rather than a feature that was added separately.

DELETE works because a commit can record removed rows without rewriting the whole partition atomically-unsafely. Time travel works because old snapshots are still valid file lists. MERGE works because the whole read-compare-write cycle can be committed as one conditional pointer move. Rollback works because "the previous version" is a value, not an archive.

The one to sit with is the time-travel query. Being able to ask what a table said at a given snapshot converts "the dashboard changed" from a memory exercise into a diff — and it is the single most useful thing this layer gives an on-call data engineer (Stale Dashboards).

Renaming a column: metadata-only for the table, not for the consumers

Before

order_id: string
order_date: date
customer_key: bigint
net_amount: decimal(12,2)
status: string

After

order_id: string
order_date: date
customer_key: bigint
net_revenue: decimal(12,2)
status: string

change net_amount renamed to net_revenue. The format tracks the column by id, so every existing data file is still readable under the new name and no bytes are rewritten.

Consumer	Effect	How it shows up
The table itself	One commit. Old files unchanged, old snapshots still resolve the column under its old name.	Loudly — it raises
A downstream SQL model selecting `net_amount`	Fails immediately at bind time with an unknown-column error. The loud, cheap case.	Loudly — it raises
A BI dashboard with the column mapped by name	The tile errors or drops the measure, depending on the tool. Usually noticed within a day by whoever reads it.	Loudly — it raises
A model using `SELECT *` then positional access	Keeps working and keeps reading the right column — until the next schema change reorders it, at which point it reads a different column with no error (Data Engineering Anti-Patterns).	Silently — no error, wrong result
A reader pinned to an older snapshot	Continues to see `net_amount`, correctly. Two consumers now disagree about the column name and both are right (Impact Analysis).	Silently — no error, wrong result

Operations that a directory of Parquet files cannot express

1-- Row-level delete: recorded in a commit, not a partition rewrite.
2DELETE FROM fct_orders
3WHERE customer_id = 'c-90218';           -- a deletion request, satisfied atomically
4 
5-- Merge from a change stream: read, match and write, committed once.
6MERGE INTO fct_orders           AS t
7USING stg_order_changes         AS s
8   ON t.order_id = s.order_id
9 WHEN MATCHED AND s.op = 'D' THEN DELETE
10 WHEN MATCHED                THEN UPDATE SET net_amount = s.net_amount, status = s.status
11 WHEN NOT MATCHED            THEN INSERT (order_id, net_amount, status)
12                                  VALUES (s.order_id, s.net_amount, s.status);
13 
14-- Time travel: the same query against the snapshot the report was built from.
15SELECT SUM(net_amount) FROM fct_orders                      ;  -- today
16SELECT SUM(net_amount) FROM fct_orders VERSION AS OF 41291  ;  -- what the report saw
17 
18-- Schema evolution: metadata only, no file is rewritten.
19ALTER TABLE fct_orders RENAME COLUMN net_amount TO net_revenue;

Syntax varies by engine and format. What does not vary is that each of these is one commit — and that a directory of files can express none of them without a window in which readers see a state that never existed.

How to build it

Most important first.

Route every write through the catalog that owns the pointer. A writer that bypasses it and drops files into the prefix has written data the table cannot see, and a cleanup job will eventually delete as orphans (The Data Catalog).
Give the table a partition specification the format can evolve, so an early mistake is correctable forward rather than a rewrite of history (The Partitioning Decision).
Choose copy-on-write for read-heavy tables with occasional corrections and merge-on-read for tables with frequent small updates — and schedule the compaction that turns the second back into the first (File Compaction).
Set snapshot expiry explicitly, as a recovery-window decision. Anything shorter than your longest-running query is a correctness bug, not just a shorter horizon (Rolling Back Data).
Run orphan-file cleanup, but only with a safety margin far longer than any in-flight write, because an orphan cleaner that runs during a slow commit deletes files a snapshot is about to reference.
Record the snapshot id a report was produced from. It turns "the number changed" from an argument into a diff (Where Did This Number Come From?).

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

Atomic visibility of a commit: the set of file changes in one commit becomes visible together or not at all, because visibility is decided by one pointer (Transactions and ACID).
Snapshot isolation for readers, which comes free from the fact that old file lists remain valid — a reader is never affected by a commit that happens after it resolved the table (MVCC: Multi-Version Concurrency Control).
Conflict detection between concurrent writers, with the losing writer failing rather than silently overwriting. This is a guarantee about writers, not a promise that retries always succeed (Optimistic vs Pessimistic).
Schema evolution with stable column identity for formats that assign column ids — a rename does not change what the data means, because the id and not the name is the key.
No guarantee against out-of-band mutation. Delete a referenced file from the bucket and the table is broken; the format assumes it owns the prefix (Data Access Control).
No cross-table atomicity. Two tables published as two commits can be observed between them, so a consumer joining them can see an inconsistent pair (Distributed Transactions).

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this

The format-specific check is a manifest audit: every data file referenced by a live snapshot exists in storage, and every file under the table prefix is either referenced by a live snapshot or older than the orphan-cleanup threshold. Both directions matter — the first catches corruption, the second catches leaked storage.
It misses everything about row content. The manifests can be perfectly consistent over files full of nulls (Data Tests).
It also misses statistics drift: if a writer records incorrect min/max values, pruning will skip files that *do* match, and the query returns fewer rows with no error anywhere. That failure is silent, engine-visible only, and the reason to be suspicious of hand-written manifests.

Freshness

Data becomes visible at commit, so freshness is set by commit cadence rather than by file arrival. A file sitting uncommitted in the bucket is not late data — it is not data at all (Late-Arriving Data).
Frequent commits produce many snapshots and many small files. The freshness gain is real, and it is paid for in planning time and maintenance load, which is a much better trade to make knowingly than to discover (Cost vs Freshness).
Merge-on-read makes writes visible sooner and pushes work to every subsequent reader until compaction. It is a freshness-for-read-cost exchange with a scheduled job on the other end of it.

When the schema or meaning changes

Add, drop, rename and reorder columns are metadata-only for id-tracking formats, so history is not rewritten and old files remain readable under the new schema (Backward Compatibility).
Type promotion is allowed in a narrow, defined set of directions — widening rather than narrowing — because the format must be able to read old files under the new type without rewriting them (Nullability & Defaults).
Partition evolution changes how *new* data is laid out while leaving old data where it is. Queries spanning the change read both layouts, which is correct and is also a performance cliff nobody warns you about (Partitioning).
None of this addresses semantics. A field that keeps its name, id and type while its business meaning changes evolves flawlessly and reports the wrong number (Semantic Changes).

How to re-run this safely

Rollback is a pointer move to a previous snapshot — the cheapest and fastest recovery operation in a data platform, and it is available exactly as long as that snapshot has not been expired (Rolling Back Data).
Backfills replace files inside a single commit, so a corrected historical partition appears atomically and the pre-fix version stays queryable for comparison (Validating a Backfill Before You Publish).
The failure with no recovery is expiring the snapshot you needed. Expiry is irreversible and its default is usually chosen for storage rather than for recovery, so it deserves an explicit decision per table (Data Retention).

What can go wrong

Failure modes

Two catalogs pointing at one table, so two writers each succeed against their own pointer and the table has two divergent histories.
Commit conflict storms: many writers targeting the same table retrying against each other, so throughput collapses without any single write failing permanently (Optimistic Concurrency Control).
Orphan cleanup deleting files belonging to an in-flight commit because its safety margin was shorter than the write.
A lifecycle rule on the bucket expiring objects a live snapshot still references — the format cannot defend against a policy applied beneath it (Storage Lifecycle).
Delete files accumulating under merge-on-read until reads carry more delete-application work than scan work, with no error and a steadily worse query time.
The mitigation failing: compaction rewriting files while a long query holds an older snapshot, which is only safe because expiry has not run — a shorter expiry turns your maintenance job into a race (Partial Failure).

Misreads

"The table format makes queries fast." It makes *planning* scale and enables file-level pruning. The scan itself is still decided by the file format, the layout and the query (The Parquet Read Path).
"It is a transaction log, so it is like a database WAL." A WAL is a durability mechanism for a single writer's recovery; this is a visibility mechanism for a version pointer. Both are logs and they solve different problems (Write-Ahead Logging).
"Time travel means we can drop backups." Snapshots live in the same bucket as the data and are bounded by expiry. A bucket-level loss or a policy misconfiguration takes both (Backup Strategy).
"Concurrent writers are solved." Conflicts are *detected*. A workload where every writer touches every partition will detect a conflict every time and make no progress, which is a correct implementation of a bad design.
"Choose the format with the most features." The choice that matters is which catalogs and engines you must support and how each format's commit protocol behaves with them. Feature parity moves; integration constraints persist.

Operating it

How you see it in production

Snapshots per table and manifests per snapshot, trended. Planning latency tracks these long before it tracks row count.
Commit attempts versus successful commits — the conflict rate, which is the honest measure of whether your writer topology is viable (Optimistic Concurrency Control).
Delete-file count and total delete rows per partition under merge-on-read, which is the compaction backlog stated in the units that matter.
Files pruned versus files read per query, which shows whether the statistics in your manifests are doing any work at all (Partition Pruning).

What changes at 10x and 100x

At 10x files, listing-based planning would have grown linearly and manifest-based planning does not, because manifests are themselves prunable by partition value. This is the property that makes the format necessary rather than merely convenient.
At 100x, the metadata tree itself needs maintenance: rewriting manifests so that each covers a coherent partition range is the difference between fast and slow planning at that size.
Writer count scales conflicts quadratically in the worst case. Beyond a handful of concurrent writers per table, partition-disjoint writes or a single serialised writer becomes the design (Orchestration).

What drives cost here

Extra read requests per query for metadata, small in absolute terms and significant on tables with pathological snapshot counts (Object Storage as Data Infrastructure).
Storage held by expired-but-not-yet-cleaned snapshots, which is the direct price of your rollback window (Storage Lifecycle).
Compaction and expiry compute, a background workload that appears in no query's cost attribution and is therefore chronically underestimated (Cost Attribution).
Against that: statistics-based file pruning reduces bytes scanned for every reader of the table, and unlike an engine-level cache that saving is shared by every engine (Scan Cost).

What this approach costs

You get transactions on object storage and you take on a metadata maintenance workload. The guarantee is genuinely new; the operational work is genuinely new too, and teams consistently plan for the first and not the second.
Merge-on-read buys write latency and costs read amplification until compaction. Copy-on-write buys read simplicity and costs a rewrite per correction. Neither is the default answer; the update frequency of the table is (Upserts and Merges).
Openness means several engines can read one copy, and it means the guarantees a consumer gets depend on the engine they used. A format is only as open as the weakest implementation you allow people to point at it.

How a table format commits

Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.

How a table format commits

Files in an object store are not a table. A table is a pointer that says which files count right now — and moving that pointer is the only atomic thing in the whole arrangement.

New objects appear in the store. No metadata references them, so no reader can see them.

	With a table format	Files under a prefix
Atomic publish	yes	no
Time travel	Reading an old snapshot is reading an old pointer.	Whatever you happened to copy elsewhere.
Row-level delete	Supported, by rewriting files or recording deletion vectors.	Rewrite the file yourself and hope nobody read it mid-way.
Planning cost	Read the manifest; prune by statistics.	List the prefix, which grows with the file count.
Cost of the choice	Metadata to maintain, snapshots to expire, and orphan files that accumulate if nothing cleans them.	None, until the first partially-visible write.

FORMAT-SPECIFICIceberg, Delta Lake and Hudi solve this differently — a metadata tree with an atomic pointer swap, an ordered transaction log, a timeline of instants — and they differ on concurrency, deletion and compaction. The shape shown here, "write files, then publish one pointer", is common to all three.

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

GENERALSnapshot indirection, manifest-based pruning and optimistic commit are shared by every format in this category. The teaching point — that a multi-object change is made atomic by reducing it to a single-object pointer swap — holds regardless of which one you run.
FORMAT-SPECIFICIceberg tracks column ids and hides partitioning behind transforms, so renames and partition changes are metadata-only; Delta records commits as ordered files in a log directory and leans on that ordering for atomicity; Hudi organises around record keys and indexes for upserts. Those choices decide which catalogs each needs and how each behaves under concurrent writers.
ENGINE-SPECIFICWhether an engine applies delete files, honours evolved schemas, or writes valid statistics varies by engine and version. A table that is correct in one engine can silently return removed rows in another, which makes engine support a correctness question rather than a convenience one.

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

Databasetransactions-and-acid mvcc write-ahead-logging isolation-levels compaction sstables lsm-trees

Cloudobject-storage object-storage-architecture storage-lifecycle

Concurrencyoptimistic-concurrency-control compare-and-swap immutability optimistic-vs-pessimistic atomicity-illusion

DSAbloom-filter binary-search

Domains that do not exist yet

— Distributed Systems owns why a conditional write on a single object is enough to linearise commits, and what a writer should do when it cannot tell whether its own commit landed.