Eight GPUs, Nine Hours
Decide what you would do from the brief alone, including whether you would change anything at all. Everything below it is available, but the exercise stops working if you open it first.
A team moved a 26-hour single-GPU training run to eight GPUs with data parallelism and got a nine-hour run and a model that scores slightly worse on validation. The request: "Get us to three hours, and find out why the model got worse. Budget is not the constraint."
Adding eight more GPUs, since budget is not the constraint. Sixteen replicas halve the per-device work and double the all-reduce participants, the input pipeline that starved eight starves sixteen, and the effective batch doubles again with the same learning rate — so the run gets marginally faster and the model gets worse again. The dashboard shows GPU utilisation at 30% and the conclusion drawn is that the framework is inefficient. The profile that would have shown the input pipeline at fault was never taken, because the proposed fix did not need one.
Read this even if you are confident. It is here rather than behind a button because it is the answer most teams actually ship, it passes review, and its cost arrives weeks later when the labels do.