Straggler Tasks
A job finishes when its slowest task does. One task out of a thousand taking twenty times as long makes the whole stage a twenty-times job, and no amount of extra capacity changes it.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
If 999 tasks finished in a minute and one is still running, how long is the job — and what would make it shorter?
Everything scheduled after this job, and every SLO expressed as "the table is ready by 06:00". A straggler is the mechanism by which a job with a comfortable average becomes a job that misses its window (The Freshness SLO).
The unit is the task: one stage's work on one partition, and the smallest thing the scheduler can place. A straggler is a single task whose duration is far above the distribution of its peers, and its indivisibility is the entire problem.
Reason about the job with averages. Total work divided by total cores gives an expected duration, capacity planning is done against that, and the schedule is set with a margin on top.
The average is fine and the maximum is not. A stage of a thousand tasks with one at twenty times the median takes roughly twenty times the median, and the average predicted something close to one (The Average Was Fine and Users Were Not).
- The average is fine and the maximum is not. A stage of a thousand tasks with one at twenty times the median takes roughly twenty times the median, and the average predicted something close to one (The Average Was Fine and Users Were Not).
- The margin absorbs the normal case and not the tail, so the job misses its window on the days that matter — month end, a backfill, a surge — and meets it every other day (Tail Latency: Why p50 Being Fine Does Not Help).
- Someone doubles the cluster after a late run. The 999 fast tasks get faster; the straggler is unchanged; the runtime is unchanged; the bill is not (Amdahl's Law).
- The straggler is on a machine that is degraded rather than a partition that is large, and every attempt to fix the data leaves it exactly where it was.
What is actually happening
- A stage ends at a barrier: the next stage needs output from every task, so the stage's duration is the maximum over its tasks rather than the mean. That is the definition, and everything else follows from it (Stages and Tasks).
- A task is indivisible. It is the unit the scheduler places and the unit the engine retries, so no amount of parallelism operates *inside* one. This is why the remedy is always to make the unit smaller or to remove it, never to add capacity around it (Work and Span).
- Stragglers come from two structurally different places. Data stragglers: this task has more, or harder, rows than its peers (Data Skew). Environment stragglers: this task has ordinary data and a slow machine — a noisy neighbour, a degraded disk, a garbage-collection pause, a throttled instance (OOM Kills and CPU Throttling).
- The two are told apart by one comparison: input bytes per task. Equal bytes and unequal durations points at the environment. Unequal bytes points at the data. This single check saves hours of misdirected work (Self Time, Total Time, and Where the CPU Went).
- Speculative execution is the engine's answer to environment stragglers: when a task runs far beyond its peers, launch a duplicate elsewhere and take whichever finishes first. It works because a different machine is genuinely faster. It does nothing for data stragglers, because the duplicate reads exactly the same oversized partition and takes exactly as long — while costing twice the resources.
The arithmetic that surprises people
A stage of a thousand tasks, each taking a minute, on a cluster with two hundred slots: five waves, five minutes, everyone happy. Now make one of those tasks take twenty minutes and change nothing else. The total work grew by less than two percent. The stage now takes about twenty minutes.
That is a four-fold increase in duration from a two-percent increase in work, and it happens because the stage ends at a barrier. The other 999 tasks finish in the first five minutes and then 199 slots sit idle watching one task, which is both the performance problem and the cost problem in a single picture.
Now double the cluster to four hundred slots. The 999 fast tasks finish in two and a half minutes instead of five. The straggler still takes twenty. The stage still takes twenty, and now twice as many cores are idle while it does. This is the moment the "just add workers" reflex is worth unlearning, because it is not merely ineffective — it is precisely, arithmetically ineffective (Amdahl's Law).
| Change | Effect on the 999 fast tasks | Effect on the straggler | Effect on the stage |
|---|---|---|---|
| Double the executors | Finish in half the time | Unchanged — a task is indivisible | Unchanged; more idle cores, higher bill |
| Faster instance type | Modestly faster | Modestly faster | Modestly faster, and expensively so |
| Speculative execution | No effect | Helps only if the cause was the machine | Helps environment stragglers, not data ones |
| Split the hot key (salting) | Slightly more tasks | Becomes several ordinary tasks | Approaches the balanced case (Salting a Skewed Key) |
| Broadcast the small side | The shuffle disappears | The concentration never happens | Best available outcome, when it applies (Broadcast Joins) |
| More partitions overall | Smaller tasks, more of them | Proportionally as large as before | Unchanged for skew; helps only oversized-everywhere stages |
1,000 tasks, 200 slots. Each bar is one wave of tasks. [SIMULATED: arithmetic on a declared model]
BALANCED |████|████|████|████|████| stage = 5 units
w1 w2 w3 w4 w5
ONE 20x TASK |████|████|████|████|████| fast tasks: 5 units
|████████████████████████████████████████| straggler: 20 units
stage = 20 units
DOUBLE THE CLUSTER |██|██|██|██|██| fast tasks: 2.5 units
|████████████████████████████████████████| straggler: 20 units
stage = 20 units, 2x the idle cores
SPLIT THE BIG TASK |████|████|████|████|████|████| the 20x work spread over 20 tasks
stage = ~6 unitsTwo causes that look identical and are not
Every straggler investigation begins in the same place and should end in one of two very different places. The symptom — a stage stuck near completion — is identical. The cause is either the data this task received or the machine it landed on, and the remedies do not overlap at all.
The discriminator is input bytes per task. If the slow task read ten times what its peers read, the data is the cause and no infrastructure change will help. If it read the same bytes and took ten times as long, the machine is the cause and no amount of key engineering will help.
It is worth building the habit of checking this first, because both remedies are expensive to apply in the wrong place. Salting a key that was never hot adds a stage and moves nothing (Salting a Skewed Key); enabling speculation against a data straggler doubles the resource cost and saves nothing.
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| max(task input bytes) / median(task input bytes) for the stage | Whether the slow task received more data than its peers. | Every form of data skew: a dominant key, a null key, a backfill concentrating a range, one enormous non-splittable file. | Rows that are equal in count and unequal in cost — an expensive branch of a user-defined function hit by a subset of rows looks perfectly balanced by bytes. |
| Executor CPU steal, GC time and disk latency for the host of the slow task | Whether the machine was healthy while the task ran. | Noisy neighbours, throttled instances, degraded disks, a heap under pressure. | A machine that was healthy on average and paused at exactly the wrong moment; per-minute aggregates hide it entirely (Coordinated Omission: When the Load Generator Lies). |
| Speculative launches versus speculative wins over a week | Whether duplicate attempts actually finish sooner than the originals. | Speculation configured against data stragglers, where duplicates never win and pure waste accumulates. | The cost of the duplicates themselves, which does not appear as a failure anywhere — it is simply capacity spent on work that was discarded. |
| Ratio of stage duration to total rows processed, tracked per run | Whether the job's duration still tracks the work it does. | The day a distribution changed, an input became non-splittable, or a machine class degraded. | A gradual drift where both grow together, which is normal growth rather than a straggler and needs a capacity answer instead (Capacity Planning: Traffic to Machines). |
Run the first two in that order. They take a minute together and they decide which entire half of the remedy space is relevant, which is worth far more than any single fix.
Speculation: what it can and cannot buy
Speculative execution is a genuinely clever mechanism and a frequently misapplied one. When a task runs well beyond the distribution of its peers, the engine starts a second copy of it somewhere else and uses whichever attempt finishes first, cancelling the other.
Against an environment straggler this is exactly right: the second machine is not degraded, so the duplicate finishes normally and the stage is rescued. Against a data straggler it is exactly wrong: the duplicate reads the same oversized partition and takes the same long time, so the job pays twice for the same delay.
There is also a correctness edge worth naming. With speculation on, a task's work may run twice, and if that task writes directly to an external system rather than through the engine's commit protocol, the write can happen twice. The engine's output commit protocol is what makes the *published* result appear once; a task that calls an external API in a loop is outside it entirely (Idempotent Data Pipelines).
What did the diagnosis say, and what does the sink tolerate?
when Input bytes are even across tasks and the slow one is on a different machine each run. The cause is the environment.
cost Duplicated work on every speculated task, and a requirement that a duplicate task attempt is harmless to the sink.
when One key holds a disproportionate share, and a composite key is still a meaningful grain for the output.
cost The output grain changes, so every downstream consumer must be checked against the new one (Grain: What Does One Row Represent?).
when The stage is a join and one side is small after filtering.
cost A size assumption that will eventually be violated as the small side grows (Broadcast Joins).
when The key is genuinely dominant, the grain cannot change, and no side is small enough to broadcast.
cost An extra aggregation stage, more partitions, and an output layout that is worse for queries filtering on the original key (Salting a Skewed Key).
when The dominant key is a single well-known entity — one tenant, one country — and it is stable.
cost Two jobs and a union instead of one job. Predictable duration, more moving parts to schedule and monitor (Orchestration).
when The straggler reads one enormous or non-splittable file.
cost A compaction or re-encode step upstream, which is work — and it fixes the problem for every future reader rather than for this job (File Compaction).
How to build it
Most important first.
- Plan and alert on the maximum task duration per stage, not the mean. The maximum is the quantity that determines whether the job lands (Percentiles: Which One, and How Many Users Is That?).
- Diagnose before remedying: compare max and median input bytes per task. It costs one glance and it decides which half of this lesson applies (Tail Latency: Why p50 Being Fine Does Not Help).
- For data stragglers, make the unit divisible — composite key, salting, a separate job for the dominant key — or remove the redistribution entirely with a broadcast (Salting a Skewed Key).
- For environment stragglers, enable speculative execution where the sink tolerates duplicate task attempts, and make sure the output commit is idempotent so a duplicated task cannot duplicate data (Idempotent Data Pipelines).
- Keep tasks numerous enough that the scheduler has slack. Several waves of tasks per core slot let a slow task overlap with others finishing; one wave exposes every straggler directly in the wall clock (Partitions: the Unit of Parallelism).
- Do not treat a non-splittable input as an acceptable default. One enormous gzip-compressed file is a permanent, unfixable straggler for every job that reads it (CSV, JSON and Their Limits).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- The engine guarantees the stage completes when every task completes. It makes no promise about how long the slowest one takes, and no scheduler can promise otherwise.
- Speculative execution guarantees only that a duplicate attempt is started; it does not guarantee it finishes sooner, and it explicitly does not help when the cause is data.
- When speculation is on, a task's side effects may occur twice. Only the commit protocol makes the published output appear once, and only for sinks that support it (Atomic Publish).
- Nothing about stragglers affects correctness. A job with a forty-minute straggler produces exactly the same rows as one without (Reconciliation).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Alert on the ratio of maximum to median task duration per stage. It is the earliest signal that a distribution or an environment has changed, and it fires long before a missed schedule does (Pipeline Metrics).
- Record task input bytes alongside duration, so that the diagnosis is already in the alert instead of requiring an investigation.
- Both miss the case where the straggler is correct and expected — a genuinely large tenant processed nightly — which is why an SLO on completion time matters more than an alert on shape (The Freshness SLO).
- End-to-end freshness is set by the sum of stage maxima, so a single straggler in a single stage propagates directly into every downstream consumer's wait (The Critical Path Is the Only Path That Pays).
- Because the effect is a tail rather than a shift in the mean, freshness SLOs written against average behaviour are systematically optimistic (Pipeline SLOs).
- Reducing the straggler is usually the single largest freshness improvement available, and it is nearly always a data or layout change rather than a capacity one.
- A schema change that adds a wide column makes every task slower and, if partitions were already uneven, makes the largest one disproportionately so.
- A source that starts emitting one enormous file instead of many smaller ones introduces a permanent straggler with no code change anywhere (File Size and the Small-Files Problem).
- Nothing about the *meaning* of the data changes here. This is a physical concern, and saying so explicitly matters: the fix belongs in layout and processing, not in the model.
- Nothing needs repairing. A job with a straggler produces correct output late, so recovery is a re-run with a different key, layout or configuration.
- When a run has already missed its window, the fastest path is usually to split the work by range or by key and run the pieces concurrently, then union (Reprocessing vs Retrying).
- If the straggler was caused by a lost executor forcing recomputation, the recovery already happened — the engine did it, and the cost was the delay (Stages and Tasks).
What can go wrong
- A single task that runs long enough to exceed the stage's timeout, failing the job after the work was almost done.
- A retried straggler paying its full duration again on each attempt, so a task that takes twenty minutes and fails three times costs an hour.
- Speculative execution duplicating a data straggler, doubling the resource cost and saving nothing (Compute Waste).
- Speculative execution duplicating a task with an external side effect, producing that side effect twice on a sink that does not deduplicate.
- The mitigation failing: adding executors, which reduces the queueing of the fast tasks and leaves the critical path exactly as long (Amdahl's Law).
- "999 of 1000 done means 99.9% complete." It means one task is the job now. Progress by task count is meaningless when durations are uneven.
- "Add workers." The remedy for a straggler is a smaller unit of work or no unit at all. Capacity addresses queueing, and a straggler is not queueing (Why Eight Cores Give You Four and a Half).
- "Speculative execution solves stragglers." It solves *machine* stragglers. Against a skewed partition it launches an identical copy of the same slow work (Data Skew).
- "The average task time is fine, so the job is fine." The average is a description of the tasks. The job is described by the maximum (The Average Was Fine and Users Were Not).
Operating it
- Max, 95th percentile and median task duration per stage, on one chart. The distance between them is the signal (Percentiles: Which One, and How Many Users Is That?).
- Task input and shuffle-read bytes at max versus median, which is what separates data from environment (Tail Latency: Why p50 Being Fine Does Not Help).
- Executor-level metrics for the machine hosting the slow task: CPU steal, garbage-collection time, disk latency (Self Time, Total Time, and Where the CPU Went).
- Speculative task launches and wins over time. Frequent launches with few wins means speculation is paying for duplicates that never help — which is the signature of data-caused stragglers.
- At 10x, a data straggler grows with the data and an environment straggler does not, so the two diverge and the diagnosis gets easier.
- At 100x, stragglers determine feasibility. A job whose critical path is one task cannot be scheduled reliably at any cluster size, so the work must be restructured (Work and Span).
- More tasks per slot helps: with several waves, a moderately slow task overlaps with others still finishing. With exactly one wave, every straggler is fully exposed in the wall clock.
- The dominant cost is idle capacity across the whole cluster for the duration of the straggler (Compute Waste).
- Speculation adds duplicated work by design; on environment stragglers it buys time with money, and on data stragglers it spends money for nothing.
- Retries of a long task multiply its cost linearly and are invisible in a successful run's summary.
- Speculative execution buys tail latency with duplicated work, and it requires a sink where a duplicated attempt is harmless. Turning it on without checking that is how a straggler mitigation becomes a data incident (Idempotent Data Pipelines).
- Smaller tasks reduce the damage any one straggler can do and increase scheduling overhead and output file count (Partitions: the Unit of Parallelism).
- Splitting a hot key into its own job gives predictable duration and doubles the number of things that must succeed for the pipeline to be complete.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALAny barrier-synchronised parallel computation has this property, from a distributed data job to a fork-join computation in one process. What differs is only the size of the unit and whether the runtime can subdivide it; none of them can subdivide a task that has already started.
- ENGINE-SPECIFICSpeculative execution and its thresholds are Spark's answer and are off or configured differently in other engines; Trino has no equivalent because it does not materialise stage output; Flink handles slow operators through backpressure rather than duplication, which changes the symptom from a stuck stage to a growing queue.
- SIMULATEDThe straggler ratios quoted here come from
src/de/sim/pipeline.tsandsrc/de/sim/layout.ts, which compute the largest partition against the mean for a declared key distribution. They are model outputs asserted byscripts/de-sim.test.ts, not timings measured on any cluster.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns the general result that a barrier across unreliable machines is bounded by the slowest participant, and the backup-request pattern that speculative execution is a batch-shaped instance of.