The Parquet Read Path
Follow SELECT country, revenue FROM events WHERE date = '2026-08-25' from a directory listing to decoded values, and count what got skipped at each of the four gates.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
For one concrete query against a partitioned Parquet dataset, exactly which bytes are read and which are never touched?
The analyst who wants the answer and the finance owner who pays for the scan. Both are served by the same thing: a reader that eliminates work at the coarsest gate it can, because everything eliminated early costs nothing at every later gate (Scan Cost).
The read path is a sequence of eliminations at four different grains — directory, file, row group, column chunk — and the whole lesson is about applying each filter at the right one. Elimination at the directory level is free; elimination after decoding is not elimination at all.
Write the query and let the engine work it out. Engines are good at this and for a well-laid-out table there is genuinely nothing to do — which is precisely why nobody notices when the layout stops cooperating.
The partition column is event_ts but the query filters on a derived date expression, so no directory is eliminated and the engine lists and opens every file in the table (Partition Pruning).
- The partition column is
event_tsbut the query filters on a derived date expression, so no directory is eliminated and the engine lists and opens every file in the table (Partition Pruning). - The dataset has one directory per hour per country. The query touches one day, and the planner still lists tens of thousands of directories before it reads any data (Partition Cardinality).
- The projection is
SELECT *inside a view that the analyst never sees, so the column-pruning gate is closed by something two layers up the SQL (Projection Pushdown). - The predicate references a column that is not the sort key, so every row group survives the statistics gate and the elimination happens after decoding — which is filtering, not skipping.
- A
LIMIT 10with no predicate makes the query look cheap and the engine still opens files until it has ten rows, which on a table with thousands of tiny files is a planning cost nobody predicted (File Size and the Small-Files Problem).
What is actually happening
- Gate one — partitions. The dataset is laid out as directories encoding a column value (
event_date=2026-08-25/). A predicate on that column is resolved against directory names before any file is opened, so entire branches disappear at zero I/O cost. This is the cheapest and largest elimination available (Partitioning). - Gate two — files and footers. Within surviving directories, the planner lists files and reads each footer. This is metadata I/O proportional to file count, which is why small files hurt even when they hold little data.
- Gate three — row groups. For each surviving file, the engine intersects the predicate with each row group's min/max statistics. Disjoint groups are skipped without fetching a byte of their data (Parquet Internals).
- Gate four — column chunks. For each surviving row group, only the column chunks the query references are fetched, using the offsets and lengths in the footer. Thirty-eight of forty columns are simply never requested (Projection Pushdown).
- Only then does decoding happen: decompress pages, decode encodings, and apply the predicate to the values that survived every gate. This last step is filtering, not skipping, and the rows it removes were already paid for.
- The gates are ordered by cost and each one is strictly cheaper than the next. Which is why the layout question is always "can I move this elimination one gate earlier" rather than "can I make the engine faster" (Predicate Pushdown).
One query, four gates
Take a concrete dataset: an events table with forty columns, partitioned by event_date, holding a year of data. And a concrete query: SELECT country, SUM(revenue) FROM events WHERE event_date = DATE '2026-08-25' GROUP BY country.
The layout below shows what the reader touches. Three hundred and sixty-four partition directories are eliminated by name, before any file is listed. Inside the surviving directory, the files are listed and their footers read. Inside those files, row groups whose statistics miss the predicate are skipped — and here nothing is skipped, because the partition predicate has already been satisfied by the directory and there is no further time filter to narrow within it.
Then gate four does the heavy lifting: two column chunks out of forty are fetched per surviving row group. That is the elimination that makes columnar storage worth having, and it is available on every query that names its columns, regardless of sort order.
- s3://lake/events/event_date=2026-08-23/one day · 6 files · skipped
- s3://lake/events/event_date=2026-08-24/one day · 6 files · skipped
- s3://lake/events/event_date=2026-08-25/part-0000.parquetrow groups 0-7 · 1 file · read
- s3://lake/events/event_date=2026-08-25/part-0001.parquetrow groups 0-7 · 1 file · read
- s3://lake/events/event_date=2026-08-25/part-0002.parquetrow groups 0-7 · 1 file · read
- s3://lake/events/event_date=2026-08-26/one day · 6 files · skipped
The interesting line is gate four: even with gate three doing nothing, thirty-eight of forty column chunks are never requested. Add AND country = 'DE' and gate three activates only if the files are sorted by country — which they are not, because they are sorted by nothing.
The gates in order, and what each one promises
CAST(event_ts AS DATE) = X into a range on event_ts and prune anyway, most cannot. Never assume the rewrite; check the plan, because the same SQL prunes in one engine and full-scans in another.Each gate is a separate mechanism with a separate failure mode, and treating them as one thing called "pushdown" is what makes debugging slow. The pipeline below states what each stage promises and how it characteristically fails.
Read the guarantees column carefully. Only gate four is unconditional — column pruning follows from the file structure and works on any data in any order. Gates one and three both depend on a physical property of the data that somebody had to arrange, and both fail silently when that arrangement is absent or wrong.
The last stage is worth its own attention. Predicate evaluation on decoded values is not a gate at all: it produces the right answer while having already paid for every byte it discards. When a query profile shows a large number of rows read and a small number returned, that is the shape of a filter that never became a skip.
- 1Partition elimination
Resolves predicates against directory names encoding a column value.
guarantees Sound elimination at zero I/O cost when the predicate references the partition column directly.
fails by Silently doing nothing when the predicate is an expression over the partition column, or references a column that is not partitioned at all.
- 2File listing and footer read
Lists surviving directories and reads each file's tail metadata.
guarantees The reader learns the schema, row group boundaries, chunk offsets and statistics of every candidate file before fetching data.
fails by Scaling with file count rather than data volume — a table of many tiny files spends most of its query time here (File Size and the Small-Files Problem).
- 3Row group elimination
Intersects the predicate with each row group's min/max statistics.
guarantees Sound: a group is skipped only when no value in it can match. Never complete — an engine may skip nothing and still be correct.
fails by Being ineffective on unsorted data, where every range overlaps every predicate, with no error and no signal (Clustering and Sort Order).
- 4Column chunk fetch
Fetches byte ranges for referenced columns only.
guarantees Unconditional — unreferenced columns are separate byte ranges and are never requested, on any data in any order.
fails by Being switched off by
SELECT *, frequently inside a view the query author did not write (Projection Pushdown). - 5Decompress and decode
Decompresses pages and decodes dictionary, run-length, delta and bit-packed values.
guarantees Exact reconstruction of the stored values. Nothing about how many of them the query needed.
fails by Becoming the bottleneck when the earlier gates eliminated little, at which point a heavier codec makes it worse (Why Analytical Data Compresses).
- 6Predicate evaluation
Applies the filter to decoded values and passes survivors on.
guarantees A correct result set. This is filtering, not skipping — the discarded rows were already fetched and decoded.
fails by Looking like success. A profile with many rows read and few returned is a layout problem wearing a correct answer.
Gates one, three and four each depend on a different decision made months earlier: partition key, sort order, and query text. Gate two depends on compaction discipline. There is no engine setting that substitutes for any of them.
Moving an elimination one gate earlier
Every optimisation in this lesson is the same move: take a filter that is currently happening late and make it happen earlier. The SQL below is the canonical example, and it is worth reading as a layout change disguised as a query change.
The important part is that the two queries return identical results. Nothing about correctness differs. What differs is which gate the elimination happens at, and therefore how many bytes were paid for before the rows were discarded.
This is also the reason SELECT * is a genuine problem in analytics rather than a style preference. It does not make a query wrong; it closes the one gate that works unconditionally, on every table, regardless of how anything was laid out (Data Engineering Anti-Patterns).
The query reads the whole table, so add workers. Runtime improves roughly with parallelism until file count or skew caps it, and the cost per query rises with the worker count.
Store the filter column, partition on it, sort within the partition on the next predicate, name the columns in the projection. The query reads a fraction of the data and the existing cluster is more than sufficient.
Adding workers divides the same work; moving an elimination earlier removes the work. The first scales cost linearly with speed and stops helping when one file, one partition or one skewed key becomes the critical path. The second reduces the bytes that any worker has to consider, so it compounds with parallelism instead of competing with it (Data Skew).
1-- LATE: everything is a filter, nothing is a skip.2-- event_date is derived at query time, so gate one cannot match a3-- directory name; SELECT * closes gate four.4SELECT *5FROM events6WHERE CAST(event_ts AS DATE) = DATE '2026-08-25'7 AND country = 'DE';8 9 10-- EARLY: the same rows, eliminated at the cheapest available gate.11-- event_date is a stored partition column -> gate one prunes directories12-- files are sorted by country within a day -> gate three skips row groups13-- two columns named -> gate four fetches 2 of 4014SELECT country, revenue15FROM events16WHERE event_date = DATE '2026-08-25'17 AND country = 'DE';18 19 20-- Verify rather than believe. Exact syntax varies by engine; the three21-- numbers to find in any of them are the same:22-- partitions pruned / files opened / row groups skipped23EXPLAIN ANALYZE24SELECT country, SUM(revenue)25FROM events26WHERE event_date = DATE '2026-08-25'27GROUP BY country;What to notice: the second query needs event_date to exist as a stored column and the files to be sorted by country. Neither is achievable by editing SQL — the query change only pays off because a layout decision was made first.
How to build it
Most important first.
- Partition on the column that appears in nearly every predicate, usually a date, and stop there. A second partition level is justified only when it is also in nearly every predicate and its cardinality is bounded (The Partitioning Decision).
- Sort or cluster within each partition on the next most selective predicate column, which is what makes gate three do anything (Clustering and Sort Order).
- Store the column you filter on, in the form you filter on it. Deriving
event_datefromevent_tsat query time closes gates one and three simultaneously. - Name the columns in the projection, all the way up through views and BI models. A pruning gate that is closed by a view definition is invisible to the person writing the query (Who Actually Consumes This Data).
- Compact so that file count is proportional to data volume rather than to batch count — gate two is the one that degrades silently as a pipeline ages (File Compaction).
- Read the query profile after every layout change. Partitions pruned, row groups skipped and bytes fetched are three separate numbers and they fail independently (Query Engines).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Every elimination is sound: a partition, row group or page is skipped only when the metadata proves no row in it can match. Pruning never changes the result set, only the cost of producing it.
- Pruning is not complete. Nothing guarantees that everything eliminable is eliminated — an engine that ignores page statistics or cannot rewrite a predicate simply reads more, correctly.
- Column pruning is guaranteed by the format's structure: unreferenced column chunks are never fetched, because their bytes are separate ranges. This one does not depend on data ordering.
- There is no guarantee of a consistent view across files. A read overlapping a write may see some new files and not others unless a table format or an atomic swap provides a boundary (Atomic Publish).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Instrument the four gates as four metrics per query: partitions pruned, files opened, row groups skipped, bytes fetched. Alert when any of them regresses for a table's top queries.
- Add an assertion after every compaction that the table's canonical query still prunes as expected, because compaction is exactly the operation that silently drops a sort order (Data Tests).
- These miss correctness entirely. A perfectly pruned query over a partition that is missing half its files returns quickly and wrongly, and gate metrics will look excellent (Missing Rows).
- The read path is indifferent to freshness, but the layout that makes it fast is not: sorted, compacted files are produced by a batch job that runs after the data lands, so the freshest partition is always the worst-pruning one.
- That asymmetry is a design choice worth stating to consumers — "queries over today cost more than queries over last week" is a true and useful sentence that almost no platform tells anyone (Cost vs Freshness).
- A query over an actively-written partition may also read files that are added mid-plan, or miss them, depending on when listing happened. Freshness at read time is therefore approximate unless a snapshot exists (Open Table Formats).
- Changing the partition column is a full rewrite of the dataset and a breaking change for anything that referenced partition paths directly. Query text stays valid; costs change completely (Partitioning).
- Adding a column does not affect the read path for existing queries — unreferenced columns cost nothing to have (Backward Compatibility).
- Renaming a filtered column closes gates one and three at once and produces a query that is correct, slow, and blamed on the engine (Breaking Schema Changes).
- The read path has nothing to recover; it is a pure read. What needs recovering is the layout, and that is a rewrite of affected partitions with validation before the swap (Reprocessing vs Retrying).
- When a compaction has destroyed pruning, the fix is to re-run it with the sort clause restored — and the way to prevent recurrence is a post-compaction assertion rather than a runbook note (Validating a Backfill Before You Publish).
- If a query is reading a partition being rewritten, the correct recovery is an atomic swap rather than a retry, because retrying against a moving directory reproduces the problem (Atomic Publish).
What can go wrong
- Gate one closed by an expression predicate, so the engine lists the whole table.
- Gate two dominated by file count after months of per-batch writes.
- Gate three ineffective because the sort key and the predicate column are different.
- Gate four closed by
SELECT *, often inside a view that the query author did not write. - A partition column with very high cardinality, where gate one is technically working and listing is now the cost (Partition Cardinality).
- An engine that does not push a supported predicate down at all — the file is fine and the plan is not (Source Pushdown).
- "The engine will figure it out." Engines optimise within the layout they are given. No planner can prune on a column that was not partitioned, sorted or stored (Query Optimizers).
- "Predicate pushdown means the predicate runs at the storage layer." It means metadata is consulted before data is fetched. The predicate is still evaluated on decoded values for everything that survived — pushdown reduces what reaches that step (Predicate Pushdown).
- "Bytes scanned went down, so the query got cheaper." Usually, but not if the reduction came from opening more files to skip more row groups; gate two costs are not measured in scanned bytes.
- "
LIMITmakes a query cheap." It bounds the rows returned, not the files considered. ALIMITwith no predicate on a many-file table is a planning-cost query.
Operating it
- Partitions pruned versus partitions considered, from the query plan. The first number to look at, always.
- Files opened per query, which is the metric that turns a small-files problem from a suspicion into a fact.
- Row groups read versus skipped from the engine profile (Parquet Internals).
- Bytes fetched from storage per query, compared against the table's total size (Scan Cost).
- At 10x data with the same layout, all four gates behave identically and total cost scales with data — which is the expected and acceptable case.
- At 100x, gate two becomes the constraint if file count grew with batch count rather than with volume, and metadata management becomes a design problem in its own right (Open Table Formats).
- At 100x concurrent queries, the planner's footer reads and directory listings become a shared bottleneck long before the data scan does, which is a completely different tuning conversation (Distributed Query Execution).
- Bytes scanned is the headline and it is set by gates one, three and four together — gate four multiplies the survivors of gates one and three, so improvements compound.
- Metadata and listing cost is set by gate two and by partition count, and it is the cost that is invisible on a bytes-scanned bill while being the reason a query takes a minute to start (What Actually Drives Data Platform Cost).
- Decode CPU is proportional to what survived all four gates, so every earlier elimination reduces it for free.
- Nothing here is a storage cost. This is the read side, and it is where the money is (Compute Waste).
- Partitioning finely makes gate one very effective and gate two very expensive. There is a crossover and it depends on your query mix, not on a rule (Partition Cardinality).
- Sorting for gate three costs a write-side shuffle and serves exactly one predicate column well.
- Compacting for gate two costs freshness on the most recent partition, which is the partition people query most.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- ENGINE-SPECIFICWhich gates an engine actually uses differs: some read page-level indexes and bloom filters, some stop at row group statistics, and predicate rewriting ability varies. The same file and the same SQL can prune at three gates in one engine and two in another.
- FORMAT-SPECIFICGates three and four are Parquet's row group and column chunk. In ORC the equivalent gates are the stripe and its row-index strides, which give finer default granularity; in a text format gates three and four do not exist at all.
- GENERALThe principle — eliminate at the coarsest gate available, because everything eliminated early is free at every later gate — holds for partitioned storage of any kind, including a partitioned operational table and a sharded index.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why listing many objects and reading many footers becomes a coordination cost rather than a bandwidth one — gate two is a metadata-scaling problem before it is an I/O problem.