Cross-Validation
k-fold cross-validation trades k trainings for a lower-variance estimate and a spread. It is the right tool for small data and model selection, the wrong tool for temporal or grouped data unless the folds respect the structure, and it is not an evaluation of the model you will ship.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
With a few thousand labelled examples, a single holdout gives a metric that moves noticeably every time the split seed changes. When does k-fold fix that, when does it make things worse, and what does it cost?
A medical device team has a few thousand labelled readings from a few hundred patients and wants to choose between three model families. Each time they re-draw the validation split, the ranking of the three changes. They have been asked which model is best and cannot give a stable answer.
Split randomly, five folds, average the metric over the folds, pick the model with the highest average. The average uses all the data for validation and all of it for training, so it is both more stable and more efficient than a single holdout.
Random folds put readings from the same patient in both training and validation. The model learns the patient's baseline in training and recognises it in validation; the cross-validated metric is an estimate of performance on patients the model has already seen, which in production is none of them (Entity Leakage).
- Random folds put readings from the same patient in both training and validation. The model learns the patient's baseline in training and recognises it in validation; the cross-validated metric is an estimate of performance on patients the model has already seen, which in production is none of them (Entity Leakage).
- The three model families are compared on the same folds and the winner is chosen. Then the same folds are used to report its performance, and the reported number is the maximum of three noisy estimates rather than an unbiased one (Evaluation Leakage).
- Feature scaling was fitted on the whole dataset before folding, so every validation fold carries information from its own rows in the scaler (Preprocessing Leakage).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a sensor reading indicates a clinically significant event. The label was assigned by a clinician reviewing the reading, with the patient's history visible.
- The decision downstream is an alert to a nurse, so the operating point matters as much as the ranking — but the immediate question is which model family to invest in.
- One example is one reading. Each patient contributes many readings, recorded across several days; readings from the same patient share a baseline and a device calibration.
- The data is small enough that a single holdout leaves too few positive events in the validation set to estimate the metric with any precision — a handful of events moving between folds changes the number visibly.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- k-fold cross-validation partitions the data into k folds, trains k models each on k−1 folds and validates on the remaining one, and reports the mean and spread of the k validation metrics. Every example is validated on exactly once; every model sees a large fraction of the data. The mean has lower variance than a single holdout because it averages over k different validation sets, and the spread is an estimate of how much the metric depends on which rows happened to be held out.
- The estimate is of the *procedure* — this model family with these hyperparameters, trained on data of this size — not of any one of the k models. The model that ships is usually retrained on all the data and has never been validated as such. That is acceptable when the procedure's performance is what the decision needs, which is the case for model selection, and a problem when a specific artifact must be certified.
- The partition decides what the estimate means. Random folds estimate performance on new rows from the same entities and the same period. Group folds — every reading from a patient in one fold — estimate performance on new patients. Temporal folds estimate performance on the future. Choosing the partition is choosing the question (Choosing a Split Strategy).
What the fold is a sample of
A validation fold is a sample from some population, and the cross-validated metric is an estimate of performance on that population. With random folds the population is "other readings from these patients during this period". With group folds it is "readings from other patients". With temporal folds it is "readings from later". Production is usually the second or third, and the default is the first.
So the first decision in cross-validation is not k. It is what the fold should be a sample of, which is the same question as what the model has to generalise across — and the answer is in the deployment, not the dataset.
1import numpy as np2 3def group_folds(groups, k, seed=0):4 """Assign whole groups to folds so no group straddles train/validation."""5 ids = np.unique(groups)6 np.random.default_rng(seed).shuffle(ids)7 fold_of_group = {g: i % k for i, g in enumerate(ids)}8 return np.array([fold_of_group[g] for g in groups])9 10def cross_validate(X, y, groups, k, make_model, metric):11 fold = group_folds(groups, k)12 scores = []13 for f in range(k):14 tr, va = fold != f, fold == f15 mu, sd = X[tr].mean(0), X[tr].std(0) + 1e-9 # fitted on training folds only16 model = make_model().fit((X[tr] - mu) / sd, y[tr])17 scores.append(metric(y[va], model.predict((X[va] - mu) / sd)))18 return np.mean(scores), np.std(scores) # the spread is the pointTwo things to notice: the scaler is fitted inside the loop, on the training folds; and the function returns the spread as well as the mean. A single number from cross-validation throws away the reason for running it.
Selection and estimation are two different uses
Cross-validation is used for two things that look alike: choosing between candidates, and estimating how good the chosen one is. The first is fine on one set of folds. The second, done on the same folds, reports the maximum of several noisy numbers as if it were one honest one — and the more candidates there were, the more optimistic the maximum.
Nesting separates them. An inner loop, on the training folds, chooses; the outer fold scores the choice. The outer estimate is then an estimate of the whole procedure including its selection, which is what will actually be run when the model is rebuilt.
Random five-fold, three model families compared, best family chosen and reported on the same folds; the nearest-neighbour-style model wins clearly.
On the first month of new patients, the chosen model is the weakest of the three; the alert rate is far above what the folds predicted.
- 1Random folds put each patient on both sides, and the winning family was the one best at recognising a patient it had already seen — which new patients defeat entirely.
- 2The reported number was the best of three on the same folds, so it was optimistic even for the population the folds measured.
- 3Feature scaling was fitted on all rows before folding, a small leak that favoured the family most sensitive to scale.
The estimate is of a procedure
None of the k models is the one that ships. The shipped model is trained on all the data, with the chosen hyperparameters, and its performance is inferred from the procedure's. That inference holds when training is stable and the hyperparameters do not depend sharply on data size; it is the assumption that makes cross-validation an evaluation at all.
It also means cross-validation says nothing about a specific artifact's bugs — a bad seed, a corrupted checkpoint, a training run that diverged. Those need a smoke test and a held-out set that the artifact itself is scored on, once.
The grouping and ordering enforced by the folds match what the deployed model will have to generalise across, so the cross-validated estimate describes production and not the dataset.
holds when Folds are grouped by patient, the population of new patients resembles the population of studied ones, and no further structure — site, device model, season — spans the groups.
breaks when The model is deployed on a new site or device the folds never separated; the patient mix shifts; readings become temporally structured in a way group folds ignore.
respond Add the new structure to the folds — group by site, or fold by time — and re-estimate before concluding the model has degraded.
How to build it
Most important first.
- Fold by the unit the model must generalise across. Here that is the patient: group k-fold, so no patient appears on both sides of any fold (Group Split).
- Put every fitted transformation inside the fold: the scaler, the encoder, the feature selector, the imputer. Each is fitted on the training folds and applied to the validation fold, or the fold leaks.
- For selection with tuning, nest it: an inner cross-validation on the training folds chooses hyperparameters, the outer validation fold scores the result. The outer estimate is then honest about the selection that happened inside it (Hyperparameters).
- Report the spread, not just the mean. Three model families whose fold-to-fold intervals overlap have not been separated by this data, and the honest answer is "any of these, pick on cost" (Metric Uncertainty).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The mean and spread of the metric across group folds, for each model family, with the transformation pipeline inside the fold. This is the number that ranks the families.
- The gap between random-fold and group-fold estimates, once, as a measurement of how much the model leans on patient identity.
- Do not report the best fold, the best seed, or the best family's cross-validated score as its expected production performance after choosing it on that score.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The grouping used for the folds is the grouping production generalises across — new patients, not new readings from known patients — and the model will not be deployed on a population that differs from the folds in some further way, such as a new device or a new site.
- The examples are exchangeable within the constraints the folds enforce: no temporal structure that a group fold ignores, no site effect that spans groups.
- The model that ships, retrained on all the data, behaves like the procedure that was cross-validated; a training instability or a data-size-dependent hyperparameter breaks that.
- Offline: run random and group folds and compare; run the group folds with the preprocessing inside and outside the fold and compare. Each gap is a leak measured.
- Online: when the model meets its first new patients, compare production performance against the group-fold estimate; a large shortfall means the folds did not capture the generalisation the deployment demanded.
- Over time: as new patients accumulate, hold them out entirely and score the shipped model — the one evaluation cross-validation never gave you.
What can go wrong
- Grouping by patient leaves some folds with almost no positive events and the per-fold metric becomes undefined or wildly unstable; stratified group folds are needed and are harder to construct (Stratified Split).
- The final model is retrained on all data with the hyperparameters chosen by cross-validation, and the training-set size difference means the chosen regularisation is slightly too strong for the larger set; the estimate is conservative, which is the good direction.
- k trainings of a slow model make the loop so long that the team runs it once with a single seed and treats the result as settled; the spread is then a spread over folds only, not over the randomness of training.
- k trainings for one estimate. For a model that trains in minutes it is free; for one that trains in hours it is the difference between iterating daily and weekly, and a single well-chosen holdout may be the practical choice.
- Group folds reduce the effective sample size — the number of groups, not the number of rows, is what the estimate averages over — and the spread widens accordingly.
- Nested cross-validation multiplies the training count again, and most teams skip it, which means most cross-validated tuning results are optimistic by an amount nobody measured.
- "Cross-validation uses all the data, so it is always better than a holdout." It estimates the procedure, not the artifact, and with random folds on grouped or temporal data it is confidently wrong in a way a single properly-structured holdout is not.
- "The five folds agreed, so the estimate is precise." Five folds with the same patient on both sides of each will agree about the same wrong thing. Fold agreement is precision around whatever the partition measures.
- "We cross-validated, so we do not need a test set." Cross-validation was used for selection. The selected model's cross-validated score is a maximum over candidates, and a separate untouched set is still needed to report an honest number (Never Tune on the Test Set).
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat k-fold estimates a procedure over k partitions and that the partition decides the question holds regardless of model family or domain.
- DATA-SPECIFICOn a few thousand rows a single holdout is too noisy and k-fold is the only way to get a usable spread; on tens of millions of rows a single holdout is already precise and k-fold buys nothing but k trainings, so the practice flips with data size.
- CONTESTEDA serious position holds that in practice a single, carefully constructed holdout — grouped or temporal as the problem demands, large enough to be precise — is more honest than k-fold, because k-fold invites random partitions, leaks preprocessing, hides the fact that the shipped artifact was never validated, and is almost never nested when used for tuning. Cross-validation's reply is that below a few thousand examples there is no holdout large enough to be precise, and the spread it reports is information a holdout cannot give.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Statistics — the bias and variance of the k-fold estimator itself, the choice of k, repeated cross-validation and the corrected resampled t-test for comparing models across folds are the statistical literature this lesson gestures at without deriving.