← IncidentsIntermediate

Ninety-Nine Tasks Done in Four Minutes, One in Fifty

Read what each party saw, commit to a cause, and only then find out which of them was right. The root cause, why the obvious reading was wrong, and the fixes are all held back until you have answered.

The symptom

What was reported, before anyone knew what was happening.

A nightly aggregation job that used to take 12 minutes now takes 54, blowing its window and delaying every downstream report. The cluster is 90% idle for the last 45 minutes of every run. Adding workers changed nothing.

What each party saw, and what each concluded

The evidence, in the form it actually arrived in — several parties, several partial views, several confident conclusions.

WhoWhat they could actually seeWhat they concludedVerdict
Scheduler512 tasks dispatched; 511 complete within 6 minutes; one still running.The job is 99.8% complete; wait for the last task.✓ right
The slow task’s workerA partition 60× larger than the median, processing at normal per-record speed.I am healthy and making progress.✓ right
Capacity engineer90% idle cluster during a job that is over its window.The job is under-parallelised; increase the worker count.✕ wrong
Data engineerTotal input volume up only 8% year on year.Growth cannot explain a 4.5× runtime increase.✓ right

1 of 4 parties reasoned correctly from what they could see and still reached the wrong conclusion. Nobody in this table is careless. Each one acted on complete-looking local information, and the information was local. That gap — between what a node can observe and what is true — is the whole domain, and one of these readings will usually be yours.

Commit before you read on

The cluster is idle, the workers are healthy, and total data volume barely moved. Commit before reading on: distinguish the two causes that produce exactly this shape — a straggler from a slow machine, and a straggler from data skew — and name the single measurement that separates them.

Write it down even if you are unsure. An unwritten guess quietly becomes “that is what I thought” the moment you read the answer.

Everything below stays hidden until you commit. Being wrong here is the exercise — the answer looks obvious the moment you read it, which is exactly why reading it first teaches nothing.