Distributed Training

Splitting a training run across many devices buys compute and pays in coordination. It is needed when the data, the model or the calendar does not fit on one machine — and for most models it is neither needed nor free.

Every worker holds the full model and a different shard of the data; each computes gradients on its shard and the gradients are averaged. It is the simplest split, and it silently multiplies the batch size.

When the model itself does not fit on one device, you cut the model rather than the data — across layers, inside matrix multiplies, or across the optimizer state. Every cut moves activations or weights over the wire, and the wire becomes the bottleneck.

Workers agree on a gradient by all-reduce — a ring exchange that is bandwidth-optimal — and the choice between waiting for everyone and not waiting decides staleness, straggler exposure, and whether two runs can ever produce the same bits.

A training checkpoint saves model weights, optimizer state, the step counter, the data position and the RNG state so a run can resume exactly where it died. It is not the model artifact, and a resume that does not restore all of it silently trains a different run.

A training run costs GPU-hours, CPU-hours, storage, network and — the multiplier that dominates — the number of times you run it. A hyperparameter search turns one run's cost into a bill, and most of the questions that reduce it are not about the hardware.