Distributed Data Processing

Splitting one computation across many machines, and the three things that buys you — memory, disk bandwidth and cores — against the one thing it costs: a network in the middle of your query.

A driver that plans and schedules, executors that hold data and run tasks, and a cluster manager that hands out machines. Almost every confusing Spark failure is explained by knowing which of the three it happened on.

A partition is the slice of rows one task processes alone. Too few and the cluster idles; too many and the scheduler dominates. And it is not the same thing as the partition in your storage path.

A stage is everything that can be done without moving data between machines. The boundary between two stages is always a shuffle, and it is always a barrier.

The one operation in a distributed job that uses the network for data. Every row is assigned a destination by key, written to local disk, fetched across the cluster and merged — which is why it dominates the runtime, the cost and the failure modes of almost every job.

Narrow: each output partition depends on one input partition, so the work stays where it is. Wide: it depends on many, so the data must move. This single distinction predicts every stage boundary in your job.

Real key distributions are not uniform. When one value holds most of the rows, the partitioner faithfully sends them all to one task — and that task becomes the job.

A job finishes when its slowest task does. One task out of a thousand taking twenty times as long makes the whole stage a twenty-times job, and no amount of extra capacity changes it.

Split the dominant key into several artificial sub-keys so its rows land in several partitions, then combine the partials. It works, it costs an extra stage — and applied to every key instead of the hot one, it does nothing at all.

When one side of a join is small enough to send everywhere, the large side never moves and the shuffle disappears. The whole technique rests on a size estimate — and on what happens when that estimate is wrong.

Transformations build a plan; nothing runs until an action asks for a result. That is what lets the optimiser see the whole query — and why your error message points at the wrong line and your pipeline ran three times.

The same question has many correct executions with wildly different costs. An optimiser turns what you asked into how it will run — using rules it can always apply and statistics it can only sometimes trust.

A stream-first distributed processor: a dataflow graph deployed once, records flowing through stateful operators, with checkpoints instead of re-runs. Compared with batch and micro-batch on what each one makes easy — not on which is better.