SIMULATED

Skew & Shuffle Lab

A distributed job finishes when its slowest task finishes. That single sentence explains why one key holding most of the rows makes a cluster twice the size run at exactly the same speed, and why the profile of a slow job is usually an idle CPU.

SIMULATEDEverything numeric on this page

Partition loads, task counts and stage timings come from models in this repository. They reproduce the mechanism — how work is distributed, and what a shuffle costs relative to a scan — not any engine's measured behaviour, which varies by version, configuration and how much memory a task was given.

These two are on one page because they are the same underlying fact seen twice. A shuffle is where data moves between tasks according to a key; skew is what happens when that key is unevenly distributed. The shuffle is what makes distributed processing possible, and the key distribution is what decides whether it is fast.

Skew

One key holds most of the rows, so one task holds most of the work. Add workers and watch nothing happen.

Skew lab — partitions, workers, and the straggler

The first thing to try is adding capacity, because it is the thing that works everywhere else. Watch it not work here, then salt the hot key and watch the runtime move.

Skew lab — partitions, workers, and the straggler
One country grows to hold most of the day. Drag the worker count and watch what does not happen.
Partitions
8sim
Largest / mean
6.31xsim
Rows on the busiest worker
3,157sim
Parallel efficiency
16%sim
Rows per partition
US3,157 · 78.9%
DE210 · 5.3%
GB183 · 4.6%
FR110 · 2.8%
IT109 · 2.7%
PL109 · 2.7%
ES89 · 2.2%
NL33 · 0.8%
Rows each worker must process
worker 13,157
worker 2210
worker 3183
worker 4110
worker 5109
worker 6109
worker 789
worker 833
The job finishes when the worker holding 3,157sim rows finishes. Every other worker is waiting. Adding another worker gives the new worker a small partition and leaves the largest one exactly where it was.
One partition holds 79% of the day. The job finishes when that task does, and adding workers changes nothing.
SIMULATEDRows per partition come from the same pipeline model the flagship lab uses. Work is measured in rows, never in seconds — a latency claim here would contradict the domain that owns latency.
How you recognise it
  • —One task is still running long after the rest finished, and the stage is waiting on exactly one of them.
  • —Adding executors changes the bill and not the runtime.
  • —The job got slow the week a large customer onboarded, with no code change involved.
  • —Retrying the failed task fails again in the same place, because the data has not changed.
What to do about it
  • —Salt the hot key only. Salting every key multiplies partitions and the mean by the same factor and moves nothing.
  • —Reconsider whether that key should be the partition key at all — the fix is often upstream of the job.
  • —Broadcast the small side of a join so no shuffle happens for it in the first place.
  • —Handle the hot key separately, as its own job, when it is genuinely one tenant rather than a distribution.
Data Skew →Salting a Skewed Key →Straggler Tasks →