GeneralisationGENERALSIMULATEDCONTESTED

Early Stopping

Stop training when validation loss stops improving, keep the best checkpoint, and accept that the validation set you stopped on is no longer an unbiased estimate of anything.

Target & data What to measure What must stay true ▶ Gradient Descent Visualizer

The problem, the obvious approach, and why it breaks

Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.

The question

Training loss is still falling. When do you stop, what do you keep, and what has the validation set become once it has decided that?

The problem

A team fine-tunes a text classifier for content moderation. Runs are scheduled for a fixed fifty epochs because that was the number in the first notebook. Some runs produce a model that is worse than the checkpoint from epoch eight, and nobody knows which runs until the model is live and the appeal rate climbs.

The obvious approach

Train for a fixed number of epochs and take the final weights. More epochs means more learning; if the loss is still going down, the model is still improving.

Why it breaks

Validation loss bottoms out at epoch eight and climbs afterward while training loss keeps falling. Epochs nine to fifty spent the optimiser fitting moderator noise. The final weights are the most-overfit weights the run produced.

How it breaks — usually after the offline metric looked fine

Validation loss bottoms out at epoch eight and climbs afterward while training loss keeps falling. Epochs nine to fifty spent the optimiser fitting moderator noise. The final weights are the most-overfit weights the run produced.
Run to run, the best epoch moves — five in one run, fourteen in another — so a fixed count is wrong in both directions: too long for the runs that converge early and too short for the ones that do not.
The final model's validation loss is logged, so the damage is visible in principle. In practice the number is glanced at, the run "finished", and the artifact is promoted because promotion is tied to run completion rather than to the best epoch (Promotion Is a Checklist, Not a Score).

Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

What is being predicted, and from what data

This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.

Target

Predict whether a post violates policy. The label comes from human moderators, with a review queue that lags posting by hours to days.
The decision is whether to hold a post for review; the operating number is precision at a review-queue capacity, and a model that has fitted moderator idiosyncrasies rather than policy loses precision on exactly the posts that matter.

Data

One example is one post with a moderator decision. Around two hundred thousand labelled posts, heavily skewed toward the allowed class, with label noise from moderator disagreement.
Training runs log loss per epoch on both the training set and a held-out validation set. Those two curves are the only evidence of what each epoch did.

How it actually works

Precisely enough to predict its behaviour — not a framework API.

Iterative training moves the weights a little each step. Early in the run, the directions that reduce training loss are the ones that also reduce validation loss: the model is learning structure shared by both sets. Later, the remaining training loss is mostly noise particular to the training set, and reducing it moves the weights in directions that raise validation loss. The validation curve turns upward at the point where the learned signal is exhausted.
Stopping there is regularisation by limiting optimisation. The weights have travelled only so far from initialisation, which for many models is equivalent to a penalty on the size of the update; the number of steps plays the role of λ. Patience — how many non-improving epochs to tolerate before stopping — handles the noise in the validation curve, and restoring the best checkpoint rather than the last one is what makes the stop useful (Checkpointing).
The cost is what it does to the validation set. Having chosen the epoch that minimises validation loss, that minimum is optimistic: among the noisy per-epoch validation numbers, the smallest was selected, and selection biases downward. The validation set has been used to make a training decision and is no longer an unbiased estimate of generalisation (Never Tune on the Test Set).

Where the two curves part

Every epoch is one pass over the training set. In the first few, both losses fall together — the model is learning what a violating post looks like. Around epoch eight the validation curve flattens and turns while training loss keeps descending. Everything after that is the model learning which specific posts in the training set were labelled which way, including the ones the moderators got wrong.

The best model in the run is the one from the turn. A fixed fifty-epoch schedule discards it and promotes the endpoint. Early stopping is the discipline of noticing the turn while it is happening, waiting long enough to be sure it is the turn and not noise, and then going back to it.

epoch   train_loss   val_loss   precision@cap   note
  2     0.412        0.398      0.61
  5     0.241        0.262      0.74
  8     0.173        0.229      0.79            <-- best checkpoint
 11     0.131        0.236      0.78            patience 1
 14     0.097        0.251      0.76            patience 2
 17     0.070        0.268      0.74            patience 3 -> stop, restore epoch 8
 50     0.011        0.402      0.63            what a fixed schedule would have shipped

The loop, with the checkpoint kept

The mechanism is a comparison, a counter and a saved copy. After each epoch, evaluate on the stopping set; if the metric improved, save the weights and reset the counter; if not, increment it; when the counter reaches the patience limit, stop and restore. Two details decide whether it works: the metric must be the one the decision uses, and the restore must actually happen.

Patience trades a little extra compute for robustness to noise in the validation curve. A patience of one stops at the first uptick, which on a small validation set is often a fluctuation; a patience of five or ten waits for evidence. The max-epoch ceiling is then a safety limit, not the schedule.

Early stopping with best-checkpoint restore

1def train_with_early_stopping(model, train_epoch, evaluate, val, patience=5, max_epochs=200):
2    best_metric, best_state, best_epoch = -float("inf"), None, 0
3    since_improvement = 0
4    for epoch in range(1, max_epochs + 1):
5        train_epoch(model)
6        metric = evaluate(model, val)          # precision at review capacity, not just loss
7        if metric > best_metric:
8            best_metric, best_epoch = metric, epoch
9            best_state = model.state_copy()    # keep the weights, not a reference
10            since_improvement = 0
11        else:
12            since_improvement += 1
13            if since_improvement >= patience:
14                break
15    model.load_state(best_state)               # restore; do NOT return the last epoch
16    return model, {"best_epoch": best_epoch, "stopping_metric": best_metric}

The returned stopping_metric is the selected maximum of a noisy sequence and is optimistic. Log it as a training artefact; report the test-split metric of the restored model as the quality number.

What the validation set has become

Before early stopping, the validation set was an unbiased estimate: rows the model had never influenced. After it, the validation set has chosen the epoch. That is a training decision made with validation data, and the number at the chosen epoch is the best of many draws — optimistic by an amount that depends on how noisy the curve was and how many epochs were compared.

This is the same rule as never tuning on the test set, one level down. Either hold a third split that touched nothing, or report the stopping number with its direction of bias stated. What is not acceptable is presenting the stopping minimum as the model's expected production quality.

Moderation classifier, stopped on validation

offline evaluation said

Stopping-set precision at review capacity of 0.79 at the restored epoch; the same model measured on an untouched test split reads 0.76.

production did

First-week precision on moderator outcomes near 0.74 — close to the test number, well short of the stopping number the team had quoted.

What explains the gap — most likely first

1The stopping number is the maximum of a noisy per-epoch sequence; selecting it biased it upward, and the test split shows the size of that bias.
2The remaining gap to production is moderator drift and a small distribution shift in post types since the training cutoff, which neither split could see.
3A smaller contribution: the stopping set shared a handful of duplicated posts with training, so its curve was slightly slow to turn and the chosen epoch a little late.

what it costs to close or detect The honest estimate costs a third split carved from a labelled set that is already the bottleneck, plus the discipline of never looking at it during development. The alternative is to quote the stopping number with a stated caveat and accept that promotion decisions carry a known optimism.

must stay trueThe stopping set still generalises

The validation set used to choose the epoch is disjoint from training at the entity level and representative of the posts the model will see, so the chosen checkpoint is the one that generalises rather than the one that memorised least visibly.

holds when Posts, threads and authors are grouped across the split; the stopping set is large enough that its per-epoch curve has a clear minimum; class balance matches production.

breaks when Duplicated or same-author posts cross the split and the stopping curve tracks the training curve; or the stopping set is refreshed with a different moderator mix and its minimum moves for reasons unrelated to the model.

how you would know The gap between the stopping-set metric and the test-split metric at the restored checkpoint, tracked per retrain; a widening gap means the stopping set has become a training set.

respond Rebuild the split with the correct grouping. Do not shorten patience or lower the epoch ceiling to compensate — that treats a leak as a schedule problem.

How to build it

Most important first.

Evaluate on validation every epoch, keep the checkpoint with the best validation metric, and stop after a patience window of no improvement. Restore the best weights, not the last (Epoch, Batch, Step).
Choose the monitored metric to match the decision — the operating-point precision, not just the loss — because the loss minimum and the precision maximum can sit at different epochs.
Hold a third split for the final estimate: train on one, stop on the second, report on the third. If the data cannot afford three splits, report the stopping-set number with the caveat that it was selected on (Train / Validation / Test).
Make the best epoch a logged output of every run so the fixed-epoch assumption cannot creep back into a config (Experiment Tracking).

What to measure

Which number actually maps to the decision — and which numbers look relevant and are not.

The test-split metric at the restored checkpoint. That is the honest number; the validation minimum is the selected one.
The distribution of best epochs across runs and retrains. A wide spread means the patience window and the max-epoch ceiling both need to be generous.
Do not measure the final-epoch metric of a fixed-length run as the model's quality. It measures the worst of the overfit tail.

What must stay true after deployment

The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.

Assumptions

The validation set used for stopping is representative of production posts and disjoint from training at the entity level, so the epoch it selects generalises rather than memorises.
The best-epoch checkpoint is the one that gets promoted and served; the artifact pipeline restores it rather than taking the final weights.
Label noise and class balance in the validation set stay comparable over time, since a noisier validation set moves the apparent minimum earlier.

How to verify — offline, online, and over time

Offline: for every run, plot both curves; confirm the restored checkpoint sits at the validation minimum and that the test-split metric is close to it. A large test-versus-validation gap is the selection bias made visible.
Online: compare precision at the operating point on the first week of moderator outcomes against the test-split number, not the validation number.
Over time: track the best epoch per retrain. A trend toward earlier stopping with a stable dataset suggests the validation set is getting noisier or the split is degrading.

What can go wrong

Failure modes in production

Patience is too short and the run stops on a noisy dip in the validation curve, before the real minimum; the model is under-trained and no one notices because "early stopping was on".
The validation set is tiny, so its per-epoch loss is noisy enough that the chosen epoch is a coin flip, and the selection bias on the reported number is large.
The validation set shares customers, threads or duplicated posts with training, so its loss keeps falling with training loss and early stopping never triggers; the model overfits with the safeguard nominally in place (Entity Leakage).

What the recommended approach costs

A third split costs data that the model would otherwise train on, which for a small labelled set is a real reduction in quality; the alternative is an optimistic number and a known direction of optimism.
Evaluating every epoch on a large validation set is compute that a fixed-length run does not spend; on a large model it can be a meaningful fraction of the training cost.
Early stopping ties the model's regularisation to the quality of a specific held-out set, so a bad validation set now produces a bad model rather than just a bad number.

Misreads

"Early stopping means the validation number is the model's score." It was the minimum of a noisy sequence, chosen because it was the minimum. Report the test split, or report the caveat.
"Training loss is still falling, so it has not converged." The training loss of an over-parameterised model falls until it fits the noise. Convergence of the training curve is not the criterion; the validation curve is.
"Fifty epochs worked last time." The best epoch is a property of the run — the data, the initialisation, the learning rate. The number that worked last time is a coincidence that early stopping exists to replace.

Where this applies

ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

GENERALAny model trained by iterative optimisation — networks, boosting rounds, iterative linear solvers — has a validation curve with a minimum that a fixed iteration count will miss in one direction or the other.
SIMULATEDThe Gradient Descent Visualizer runs real descent on synthetic surfaces; its loss curves show why more steps are not monotonically better, but its numbers are the mechanism, not a measurement on a moderation dataset.
CONTESTEDA defensible position holds that the third split is a luxury: for a model retrained monthly, the optimism from stopping on validation is small, stable and the same direction every time, so the validation number is a perfectly usable relative measure and the data is better spent training. That is right when the validation set is large and the decision is "did this retrain improve on the last one"; it is wrong when the number is used to promise absolute quality to a stakeholder.