Partitions: the Unit of Parallelism
A partition is the slice of rows one task processes alone. Too few and the cluster idles; too many and the scheduler dominates. And it is not the same thing as the partition in your storage path.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
How many pieces should this data be cut into, and what decides the number — the data, the cluster, or the query?
Anyone who has to fit a job inside a window, and every reader of the files it writes. Partition count at the end of a job becomes file count in the output, so a compute decision silently becomes a layout decision for everyone downstream (File Size and the Small-Files Problem).
One partition is a set of rows that one task holds in memory and processes without coordination. The right size is bounded below by scheduling overhead — a task that finishes in milliseconds was not worth dispatching — and above by the memory one task is given.
Accept whatever partition count the engine produced. It picked a number from the input files, it picked another number for the shuffle, and both defaults exist for a reason. For a job in the middle of the size range this is entirely fine.
The input is four large compressed files in a non-splittable codec. The engine produces four partitions, four tasks run, and a two-hundred-core cluster uses four cores (CSV, JSON and Their Limits).
- The input is four large compressed files in a non-splittable codec. The engine produces four partitions, four tasks run, and a two-hundred-core cluster uses four cores (CSV, JSON and Their Limits).
- The shuffle partition count is a fixed default while the data grew tenfold. Each post-shuffle partition no longer fits in task memory, every task spills, and the stage is now bound by local disk (Disk and Storage: Latency, Throughput, IOPS and the fsync Tax).
- The same default applied to a tiny job produces hundreds of partitions holding a handful of rows each, and the job spends more time dispatching tasks than computing. The output is hundreds of small files, which every future reader now pays for (File Compaction).
- Someone repartitions by a high-cardinality column to "increase parallelism". The repartition is itself a full shuffle, and it produces one partition per distinct value — most of them nearly empty and a few of them enormous (Data Skew).
What is actually happening
- Partition count at the start of a job comes from the input: how many files there are, whether they can be split, and the row-group or block boundaries inside them. Columnar formats are splittable at row-group granularity, which is one of the quieter reasons they matter here (Parquet Internals).
- Partition count after a shuffle is a separate number, chosen by configuration or by the planner rather than by the input. This is the number that decides how much data each reducing task holds, and it is the one most often left at a default that was appropriate for a different dataset.
- Each task is dispatched, its closure deserialized, its work run, and its result accounted for. That fixed cost is small and it is per task, so it matters entirely in proportion to how little each task does (Parallel Overhead).
- The number of tasks that can run at once is the total core slots across executors. Partitions beyond that number queue, which is good — a queue keeps every core busy and lets fast tasks and slow tasks even out. Fewer partitions than slots means idle cores that no configuration can fill (Worker Pools Beyond Threads).
- A partition is also a memory budget. The task must hold its rows, its aggregation state and its shuffle buffers inside the memory its executor allocated per slot, and exceeding that means spilling to disk rather than failing — which is why the symptom of too-large partitions is slowness rather than an error.
Two words called partition, and they are not the same thing
This is the single most common confusion in this part of the field, it is genuinely the same word for two different things, and it produces real damage — so it is worth being blunt about the difference before anything else.
A storage partition is a directory in a table's path, chosen so that a query with a matching predicate can skip everything else without opening it. It is a physical layout decision, it persists between jobs, it is chosen from the query patterns, and its cardinality must stay low enough that each partition holds a substantial amount of data (Partitioning).
A compute partition is a transient in-memory slice of rows that one task processes. It exists for the duration of a stage, its count is chosen from the cluster size and the data volume, and it means nothing to anyone after the job ends. Numbers in the thousands are perfectly normal.
The damage happens when the two are conflated in one direction: someone partitions the *table* by a high-cardinality column such as user_id because they want more parallelism, and creates millions of directories holding a handful of rows each. Parallelism did not improve — that comes from splits and shuffle counts — and the table now has a metadata problem that only a rewrite fixes (Partition Cardinality).
| Storage partition | Compute partition | |
|---|---|---|
| What it is | A directory in the table path, e.g. dt=2026-08-25/ | A slice of rows held by one task in memory |
| Lives for | As long as the table does | The duration of one stage |
| Chosen from | The predicates queries actually carry | Cluster core slots and data volume per stage |
| Good cardinality | Low — tens to low thousands, each holding real data | High — typically several per core slot, thousands is normal |
| What it buys | Reading less: whole directories skipped unopened (Partition Pruning) | Doing more at once, and retrying failures in small units |
| Failure when wrong | Millions of tiny files, metadata-dominated queries, a rewrite to fix | Idle cores or scheduling overhead — fixed by re-running with a different number |
| Who is affected | Every reader of the table, forever | This job, this run |
Too few, too many, and the band in between
Partition count has a wrong answer on both sides, and the two wrong answers look completely different in the cluster UI. Too few shows as a stage with a handful of long-running tasks and a mostly idle cluster. Too many shows as a stage with thousands of tasks each finishing almost immediately, where the total time is dominated by dispatch.
The useful target is a band rather than a value: enough tasks that every core has several to work through — so that a slow task is absorbed rather than exposed — and each task large enough that its fixed cost is negligible against its work.
The band moves. It moves when the data grows, when the cluster is resized, and when a column is added that makes each row wider. This is why partition count is a thing to observe rather than a thing to set once, and why the output file profile is a good proxy to watch even if you never look at the job.
Set the shuffle partition count to a fixed value in the job's configuration, because that value made a job fast in 2023. Never revisit it. As the data grows, each task holds more, spills, and the stage slowly becomes disk-bound — with no error, no alert and no code change to blame.
Choose the count so that each post-shuffle partition holds an amount of data a task can process in memory, and so that the resulting task count is several times the cluster's core slots. Track per-task shuffle-read bytes and spill bytes over time, and adjust when they drift. Repartition explicitly before writing, so the output file layout is chosen rather than inherited.
Partition count is a relationship between three things that all change independently — data volume, row width and cluster size — so any fixed value is correct only at the moment it is chosen. Watching spill and per-task bytes turns a slow decay into a visible signal, which is the difference between tuning once and tuning forever.
Cluster: 20 executors x 4 cores = 80 task slots partitions = 12 ████ 68 slots idle, tasks huge, spilling partitions = 80 ████████████████████ one wave, zero slack for a slow task partitions = 240 ████████████████████ x3 three waves, unevenness absorbed partitions = 40000 ████ ... ████ (tiny) dispatch dominates; 40k output files The band is "several waves of substantial tasks", not a number.
Where partition counts actually come from
Three separate mechanisms decide how many partitions a job has at different moments, and they are frequently confused with one another. Knowing which one is in force at the stage you are looking at is most of the diagnosis.
At the source, the count comes from the input layout: file count, splittability, and row-group boundaries. No configuration raises it above what the files allow, which is why a job reading four gzip-compressed CSV files cannot use more than four cores no matter how large the cluster is (CSV, JSON and Their Limits).
After a shuffle, the count is whatever the shuffle was told to produce. This is a free choice and the most consequential one. Before writing, the count is whatever the final stage has — which is why an explicit repartition immediately before the write is the standard way to control output file count without disturbing the rest of the job.
- 1Source split
Divides input files into readable chunks by row group, stripe or block.
guarantees Every row is read exactly once, across a number of partitions bounded by what the files permit.
fails by Non-splittable compression capping parallelism at the file count, invisibly and permanently for that input.
- 2Narrow chain
Runs filters and projections within each partition; the count does not change.
guarantees Partition count is preserved through every narrow transformation (Narrow and Wide Transformations).
fails by A highly selective filter leaving thousands of nearly empty partitions that still cost a task each.
- 3Coalesce
Merges adjacent partitions without moving data across the network.
guarantees No shuffle, and no redistribution — the surviving partitions are unions of neighbours.
fails by Reducing parallelism for the whole upstream chain when applied too early, because there is no shuffle to absorb it.
- 4Shuffle / repartition
Redistributes rows by key or round-robin into a chosen number of partitions.
guarantees Rows with the same key share a partition; the count is exactly what was requested.
fails by Producing wildly uneven partitions when the key is skewed, whatever the count (Data Skew).
- 5Write
Each partition of the final stage writes its own file per output location.
guarantees One file per partition per storage partition — which is precisely why this count is a layout decision.
fails by Inheriting a large shuffle count and emitting tens of thousands of small files (File Size and the Small-Files Problem).
The last row is the one that leaks outside the job. Every other partition-count decision is forgotten when the application exits; this one is handed to every future reader.
How to build it
Most important first.
- Aim for enough partitions to give every core several tasks, so that scheduling can absorb unevenness, and few enough that each task has substantial work. The ratio matters more than the absolute number, and it is a property of the cluster as much as of the data (Sizing a Thread Pool).
- Set the post-shuffle partition count from the shuffled data volume rather than from a default, and revisit it when the data grows. It is one of the few configuration values that genuinely repays attention (The Shuffle).
- Coalesce rather than repartition when reducing partition count, because coalescing merges neighbouring partitions without a shuffle while repartitioning redistributes everything across the network.
- Control the output file count deliberately at the end of the job. The last stage's partition count is the file count you hand to every downstream reader, and the right number there is a layout question, not a compute one (File Size and the Small-Files Problem).
- Never choose a partition count to fix skew. Splitting an uneven distribution into more pieces leaves the largest piece exactly as large relative to the mean (Salting a Skewed Key).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Every input row belongs to exactly one partition, and partitions are processed independently. That is the entire promise, and it is what makes retry and parallelism possible.
- Nothing guarantees partitions are equal in size. Their sizes come from the data and from the partitioner, and evenness is a property you check rather than receive (Data Skew).
- Nothing guarantees a stable mapping from row to partition across runs, unless the partitioner is deterministic and the partition count is fixed. Changing the count changes the mapping for every key (Topics and Partitions).
- Partition boundaries carry no semantic meaning. Reasoning that assumes a partition corresponds to a day, a customer or a file is reasoning about an implementation detail (Determinism: Same Input, Same Output?).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Record the per-partition row count distribution for the shuffle stages of important jobs, and alert when the maximum diverges from the median beyond a threshold. This catches a distribution change before it becomes a missed window (Volume Anomalies).
- Check the output file count and size profile after the job. It is the cheapest available detector of a partition count that has drifted away from the data volume (File Size and the Small-Files Problem).
- Neither says anything about correctness. A job with perfectly even partitions and ideal file sizes can produce entirely wrong numbers, and this module has no check that will tell you otherwise.
- Partition count sets how much of a job can proceed at once, so it directly affects how long the job takes and therefore how fresh its output is. It cannot make the output fresher than the schedule that triggers it.
- Too many partitions inflate the fixed portion of the runtime, which is felt most in frequently scheduled small jobs — precisely the jobs whose freshness people are trying to improve.
- A stage whose partitions do not fit memory spills, and a spilling stage has a runtime that is no longer proportional to its data. That is where a job that used to finish inside its window stops doing so.
- Data volume growth silently invalidates a fixed shuffle partition count. Nothing announces it; the job simply starts spilling one day and gets slower every week after.
- Adding a column widens every partition, which changes how many rows fit in a task's memory even though the row count is unchanged (Projection Pushdown).
- Changing the partition count changes which key lands in which output file, and any downstream consumer that relied on the previous arrangement — a bucketed join, for instance — silently stops benefiting from it (Bucketing).
- Partitioning has no persistent state, so recovery is simply re-running with a better number. Nothing is corrupted by a bad choice; the cost is time and money rather than data.
- The exception is output: a run that produced a hundred thousand tiny files has left a mess that a downstream compaction has to clean up, and until it does, every reader pays (File Compaction).
- Re-running with a different partition count produces a different physical arrangement of the same logical result. Any consumer comparing file listings between runs will see a difference that means nothing (Atomic Publish).
What can go wrong
- Too few partitions: idle cluster, tasks too large for memory, spilling, and a job whose duration is set by a handful of tasks.
- Too many partitions: scheduling overhead dominates, and the output is a small-files problem for everyone downstream.
- A repartition intended to increase parallelism that instead introduces a full shuffle and a skewed distribution.
- Non-splittable input capping parallelism at the file count, which no configuration can raise (CSV, JSON and Their Limits).
- The mitigation failing: raising the partition count on a skewed dataset, where the largest partition stays proportionally as large as it was.
- "More partitions means more parallelism." Only up to the number of core slots. Beyond that it means more queueing and more overhead, and past a point it means a small-files problem (Why Eight Cores Give You Four and a Half).
- "A compute partition and a storage partition are the same thing." They are not, they are chosen by different criteria, and confusing them produces a table partitioned by
user_idin the name of parallelism (Partition Cardinality). - "Repartitioning is cheap." Repartitioning is a shuffle. It is one of the most expensive operations available and it is frequently added to a job in the belief that it is a hint (The Shuffle).
- "The engine picks the right number." It picks a number from the input layout and a default. Both are reasonable starting points and neither knows what your data grew into.
Operating it
- Tasks per stage against total core slots. The ratio tells you immediately whether the cluster can be busy at all (Saturation: The Reading Utilization Cannot Give You).
- Per-task input and shuffle-read bytes, as a distribution. The median tells you about sizing; the maximum tells you about skew (Tail Latency: Why p50 Being Fine Does Not Help).
- Spill bytes per stage, which is the direct signal that partitions no longer fit their memory budget (Memory Pressure, Swap and the OOM Killer).
- Output file count and mean file size per run, tracked over time, because this is the number that drifts without anyone changing anything.
- At 10x data with the same partition count, each task holds ten times as much and spilling begins. The count has to grow with the data; it is not a constant.
- At 100x, partition metadata itself becomes a driver-side cost, and jobs are frequently split by range rather than run as one enormous application (Incremental Processing).
- More executors change the ideal count, because the target is several tasks per slot and the slot count moved. This is the sense in which the number is a property of the cluster and not only of the data.
- Idle cores are the cost of too few partitions, and they are billed exactly like busy ones (Compute Waste).
- Scheduling and serialization overhead is the cost of too many, and it grows linearly with task count regardless of how little each task does.
- The downstream cost of the resulting file layout is usually larger than either, and it is paid by every query for as long as the files exist (Scan Cost).
- Tuning partition count for a job's runtime often produces an output file layout that is wrong for its readers. The two objectives genuinely conflict, and the usual resolution is to repartition just before writing.
- More partitions give better failure granularity and better load balancing, and cost fixed overhead per task. There is no setting that is best on both axes.
- Explicit control means one more number that ages. A hard-coded partition count is correct on the day it is written and slowly stops being correct thereafter.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- ENGINE-SPECIFICSpark exposes partition count directly and expects you to manage it; Trino derives splits from the connector and manages parallelism itself; a serverless warehouse hides it entirely and gives you slots or credits instead. The physics is identical, but only in the first case do you set the number.
- FORMAT-SPECIFICSplittability decides the floor on input partitions. Parquet and ORC split at row-group or stripe boundaries, Avro at sync markers, and a gzip-compressed CSV cannot be split at all — so with the last one, file count is a hard cap on parallelism no configuration can raise.
- SCALE-SPECIFICThe advice to tune partition count matters above the point where a stage no longer fits comfortably in memory. Below it, defaults are fine and the tuning effort is better spent on layout, which pays every query rather than one job.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why a fixed partitioner makes the row-to-partition mapping change when the partition count changes, and what that costs a system that assumed the old mapping.