Partition & File Format Explorer
Where bytes physically sit decides what a query is obliged to read. These are the highest-leverage decisions in an analytical platform and the least visible ones — nothing in a query plan says “this table was partitioned badly two years ago”.
File counts, row counts and the read/skip decisions below come from src/de/sim/layout.ts, a model that reproduces how pruning works rather than what any particular engine measured. Which of these skips a given engine actually performs varies, and the model says so.
Two independent skips compound. Partitioning decides which files the reader opens; the file format decides how much of each opened file it has to decode. A dataset that gets both right reads a small fraction of itself to answer a question; a dataset that gets either wrong reads all of it and looks, from the outside, like a dataset that needs a bigger cluster.
The failure mode worth internalising is that neither skip is guaranteed by declaring it. A partition key the predicate does not mention, a filter wrapped in a function the planner cannot see through, or a SELECT * all silently return you to reading everything — with the layout still in place, still described in the documentation, doing nothing.
Partition Explorer
Choose a partition key and a predicate, and see which paths the reader has to touch — and which ones it can ignore without opening.
Watch two failure modes rather than one. A key that is too coarse makes every query read too much; a key that is too fine moves the cost into metadata, and listing the table starts costing more than reading it.
- —It appears in the
WHEREclause of the queries that actually run, unwrapped and comparable. - —Its cardinality is low enough that the partition count stays in a range the catalog can list cheaply.
- —It divides the data reasonably evenly, so no single partition holds most of the work.
- —It is stable — a value that gets corrected after the fact means rewriting partitions rather than appending to them.
- —A high-cardinality key turns a table into millions of tiny objects, where the per-request cost dwarfs the data.
- —A skewed key puts most of a job on one task, and adding workers moves nothing.
- —Changing the key later means rewriting the dataset and every query that filters on the old one.
- —A partition scheme nobody documented becomes a scheme nobody dares change.