Physical Data Layout
Where bytes physically sit decides what a query must read. File size, compaction, partitioning, pruning, cardinality, clustering and bucketing — the highest-leverage and least-visible decisions in analytics.
Two datasets with identical rows and identical schemas can differ by an order of magnitude in what a query must read. The difference is which rows share a file and which files share a directory.
The same bytes split into a million objects behave nothing like the same bytes in a hundred. On object storage the cost is per request, so this is a problem about file count, not data volume.
Rewriting many small files into fewer larger ones. What it buys, what it costs, and what a reader sees if it is halfway through when their query starts.
Putting a column's value into the directory path so a reader can exclude data without opening it. The cheapest skip available, and the only one that costs nothing to evaluate.
The planner eliminating partitions before reading. It is a best-effort behaviour, not a guarantee — and there are four common predicate shapes that silently defeat it while looking completely correct.
A partition key with too many distinct values produces more metadata than data. The target is enough rows per partition to justify a file, and few enough partitions to list.
Within a partition, the order rows were written in decides how selective file statistics are. Sorting is what turns a min/max into a skip, and it decays as soon as you stop maintaining it.
Hashing a key into a fixed number of files so that rows with the same key always land in the same bucket. One physical technique, and it earns its keep almost exclusively when it lets a join skip the shuffle.
Five questions that decide a partition key: what the filters carry, how many distinct values, how much data per partition, how often data is appended, and whether the distribution is skewed. Answer them from data, not from intuition.