Three Splits, One Job Each
Decide what you would do from the brief alone, including whether you would change anything at all. Everything below it is available, but the exercise stops working if you open it first.
A colleague hands over a notebook: "Train/test split, 80/20. I tried about forty settings and kept the one with the best test accuracy. Test accuracy 0.91 — should be the number in the launch doc." There is no validation set.
Re-running the forty settings with five-fold cross-validation over the whole dataset and reporting the best cross-validated score. It looks rigorous — five folds instead of one split, every example used — and it removes the one thing the process had going for it, a set that had never been used for selection. The best-of-forty cross-validated score is still a maximum over noisy draws, and now there is no untouched data left to expose it. The launch number goes up, the confidence in it goes up, and the production number comes in below both.
Read this even if you are confident. It is here rather than behind a button because it is the answer most teams actually ship, it passes review, and its cost arrives weeks later when the labels do.