Preprocessing Leakage
A scaler, imputer, encoder or feature selector fitted on the full dataset before the split has seen the validation rows. The order of operations is the leak.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
No feature knows the future and no entity straddles the split. How can a normaliser leak, and why is the fix a matter of ordering rather than of columns?
A team built a clinical risk model on a small dataset — a few hundred patients, a few thousand candidate lab measurements. They selected the fifty most label-correlated features, standardised everything, ran cross-validation and reported a strong result. A reviewer re-ran the pipeline with the selection inside the folds and the result was barely better than chance.
Clean the data first, then model it. Imputation, scaling and feature selection are preprocessing; they happen before the modelling step, on the whole dataset, so everything downstream sees consistent columns.
Feature selection on the full dataset used every validation label to pick the fifty columns. Inside each fold the classifier is trained on features chosen partly because they correlate with the labels it is about to be tested on. The cross-validated number is a measurement of that selection, and with thousands of candidates and a few hundred rows, chance correlations are plentiful.
- Feature selection on the full dataset used every validation label to pick the fifty columns. Inside each fold the classifier is trained on features chosen partly because they correlate with the labels it is about to be tested on. The cross-validated number is a measurement of that selection, and with thousands of candidates and a few hundred rows, chance correlations are plentiful.
- The mean imputer and the scaler used validation rows to compute their statistics. On this dataset the effect is small; on a skewed or tiny dataset it is not, and either way the validation rows have shaped the transformation applied to them.
- In production, the pipeline's statistics were computed once from the full training data and shipped, so serving sees no additional leak — but the reported number was never a property of what shipped. The honest number, from selection inside the folds, was what production delivered.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict from a panel of measurements whether a patient has a condition. The label is a confirmed diagnosis.
- With few patients and many candidate features, the risk is that the pipeline finds features that are correlated with the label by chance on this sample.
- One example is one patient with several thousand numeric measurements, many of them missing, on different scales, plus a handful of categorical fields.
- The pipeline is: impute missing values with the column mean, standardise each column, select the top features by correlation with the label, then cross-validate a classifier on the selected, scaled features.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Any step that is *fitted* — learns parameters from data — is part of the model. A scaler learns a mean and a standard deviation; an imputer learns a fill value; an encoder learns a vocabulary and, for target encoding, per-category label rates; a feature selector learns which columns to keep. If the fitting sees the validation rows, the model has seen the validation rows.
- The size of the leak scales with how much label information the step extracts. A scaler extracts none directly — only the feature distribution — so its leak is a small contamination that mostly matters on tiny or skewed data. A supervised feature selector extracts a great deal, and with many candidates and few rows it can manufacture an entire spurious model. Target encoding sits in between and is treated on its own in Target Encoding.
- The simulator's
global-normalizationinjection shows the small end honestly: standardising on all rows before a group split barely moves the gap, because the map is affine and the model is linear. The lesson is not that it is harmless but that its size is a property of the data and the model, which the contaminated evaluation cannot tell you. - The rule that follows is about order: split first, fit every preprocessing step on the training fold only, apply the fitted step to the validation fold. In cross-validation, this happens inside every fold, which means the preprocessing is refitted k times (Cross-Validation).
Fit on train, apply to the rest
The wrong order reads naturally: clean, then split, then model. The right order is less natural and is the one that matters: split, then fit every cleaning step on the training side, then apply those fitted steps to the validation side without refitting. The difference is invisible in the output columns — both produce standardised numbers — and entirely visible in what the validation number means.
The mechanism is that a fitted step carries information from the rows it was fitted on into the transformation it applies. For a scaler that information is a mean and a spread; for a selector it is which columns correlated with the labels. Fitting on the full dataset moves that information from the validation rows into the model, and the evaluation then rewards the model for having it.
Impute, scale and select on all rows; then split into train and validation; then train and evaluate. Every statistic, and every selected column, has seen the validation rows and their labels.
Split first. Fit the imputer, scaler and selector on the training rows only. Transform the validation rows with those fitted objects. In cross-validation, do this inside every fold.
The validation rows must be unseen by everything that learns parameters, and preprocessing steps learn parameters. Only then does the validation number describe how the whole pipeline — not just the classifier — will behave on data it has not seen.
1# WRONG: statistics and selection see the validation rows2mu, sd = X.mean(0), X.std(0)3Xs = (X - mu) / sd4keep = top_k_by_label_correlation(Xs, y, k=50) # touches every label5X_tr, X_va, y_tr, y_va = split(Xs[:, keep], y)6model.fit(X_tr, y_tr); score(model, X_va, y_va) # contaminated7 8# RIGHT: split first; fit everything on the training fold only9X_tr, X_va, y_tr, y_va = split(X, y)10mu, sd = X_tr.mean(0), X_tr.std(0) # training statistics11keep = top_k_by_label_correlation((X_tr - mu) / sd, y_tr, k=50)12model.fit(((X_tr - mu) / sd)[:, keep], y_tr)13score(model, ((X_va - mu) / sd)[:, keep], y_va) # honest14# ship mu, sd and keep with the model; serving applies them, never refitsThe selection line is the one that changes the result on small data. The scaling lines change it a little. Both are wrong in the first block for the same reason, and the fix costs nothing but the ordering.
Small leaks and large ones
The simulator's global-normalization case is deliberately undramatic: standardising on all rows before the split barely moves the validation/future gap, because a linear model is almost indifferent to an affine map and the extra rows change the mean very little. That is honest, and it is the point. The refusal to fit on the full dataset is not justified by the leak always being large; it is justified by the size being unknowable from inside the contaminated evaluation and the fix being free.
At the other end, supervised selection on a small, wide dataset can be the entire result. With thousands of candidate columns and a few hundred rows, dozens will correlate with the label by chance; select those, cross-validate a classifier on them, and the folds confirm the correlation they were used to find. The shuffled-label test exposes this in one run.
looks like A sensible dimensionality reduction step before modelling, producing a clean feature table that the cross-validation loop then consumes.
why it leaks The selection read every label, including those of the rows each fold will validate on. Columns were kept partly because they correlate with the validation labels by chance; the classifier inherits that correlation and the fold confirms it.
fix Move selection inside the fold; run the shuffled-label test; hold out a test set before any fitting and report its number once.
Cross-validated metrics well above chance with fifty features selected by label correlation on the full dataset before the folds were formed.
On new patients from the same clinic the model performs close to the honest in-fold pipeline, which is close to chance; the reviewer's re-run with selection inside the folds predicted this before any patient was scored.
- 1Selection outside the folds used every validation label; with thousands of candidates and a few hundred rows, dozens correlate with the labels by chance, and the folds confirmed the correlation they were used to find.
- 2Mean imputation and scaling on the full dataset added a small further contamination that on this data barely registers but was fitted on validation rows all the same.
- 3A small residual gap may be genuine population shift between the study cohort and new patients, which a shuffled-label test would not attribute to the pipeline.
The artifact is the pipeline
The ordering guarantee survives only if the fitted preprocessing is part of the model object that ships. A scaler fitted on the training fold and stored in a notebook variable is correct today; a serving service that recomputes the mean from last week's traffic is a different transformation with the same name, and the leak has been traded for skew.
So the artifact carries the statistics, the vocabulary and the selected columns, and the serving path applies them without refitting. A contract test at deploy time comparing the artifact's statistics to the training run's logged values is cheap and catches both the refit and the stale-artifact case (Serving Contract Tests).
Every parameter of every preprocessing step was learned from the training fold alone, is stored in the artifact, and is applied unchanged in validation, test and serving.
holds when Preprocessing is a fitted pipeline object inside the artifact; cross-validation refits it per fold; serving loads it and only transforms.
breaks when A column list derived from a full-data exploration is hard-coded; serving recomputes statistics from traffic; a rebalancing or deduplication step is applied before the split; a new preprocessing step is added outside the pipeline object.
respond Move the step inside the pipeline object, refit inside folds, re-report against a test set held out before any fitting, and add the contract test so it does not regress.
How to build it
Most important first.
- Make preprocessing part of the model object: one pipeline that is fitted once on the training fold and applied, unchanged, to validation, test and serving (Preprocessing Lives in the Artifact). If the scaler is not in the artifact, the ordering guarantee is a convention that a refactor will break.
- In cross-validation, fit the whole pipeline inside each fold. Any step that touches the label — selection, target encoding, class-balancing — must be inside the fold without exception.
- Treat unsupervised steps with the same discipline even when the leak is small, because the cost is a line of code and the size of the leak on the next dataset is unknown.
- Ship the fitted statistics with the model and read them at serving time; never recompute the mean from serving traffic (Train / Serve Skew).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The metric with preprocessing fitted inside the folds versus fitted on the full dataset. The gap is the leak; for supervised selection on small data it can be the whole result.
- For the shipped model, the metric on a test set that was held out before any fitting at all — the only number the artifact's statistics have never touched.
- A cross-validated number from a pipeline where selection happened outside the folds is not a measurement of anything.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every fitted preprocessing step was fitted on the training fold alone, and the statistics it learned are stored in the artifact and applied unchanged in serving.
- No column list, vocabulary or selected-feature set that was derived from the full dataset survives as a constant in the pipeline.
- The serving path does not refit any transformation on serving traffic.
- Offline: re-run the evaluation with every step inside the folds and compare to the reported number; for any step that touches labels, the comparison is mandatory.
- Offline: a shuffled-label test — permute the labels and rerun the whole pipeline including selection; if the metric stays well above chance, the pipeline is manufacturing signal from the evaluation rows.
- Online: assert the scaler and imputer statistics in the serving artifact match the training run's logged values, as a contract test at deploy time.
What can go wrong
- The training code is fixed to fit inside folds, but the exploratory notebook that chose the fifty features on the full dataset still exists, and its column list is hard-coded into the pipeline. The selection leak survives as a constant.
- The pipeline is correct offline, but the serving path re-standardises using statistics from a recent batch of traffic, so training and serving apply different maps — the leak was removed and skew was introduced.
- Class rebalancing by oversampling is applied before the split, so duplicated minority rows appear on both sides — preprocessing leakage that is also entity leakage.
- Refitting preprocessing inside every fold multiplies the cost of cross-validation by the cost of the preprocessing, which for expensive steps such as a large encoder can dominate.
- Fitting on the training fold only gives slightly noisier statistics than fitting on all data, which on tiny datasets is a real loss — but the alternative is not knowing the number is honest.
- Bundling preprocessing into the artifact couples the serving system to the training framework's serialisation, which is a deployment constraint.
- "Scaling cannot leak; it does not use the label." It uses the validation rows' distribution, which is a small contamination for a scaler and a large one for other unsupervised steps such as PCA or clustering on small data. The rule is about fitted steps, not about labels.
- "We used cross-validation, so the number is honest." Cross-validation is only honest if everything fitted is refitted inside each fold. Selection outside the folds makes k-fold as leaky as a single split.
- "The leak was tiny in the simulator, so global normalisation is fine." Its size there is a fact about that data and that model. The reason to split first is that the size is unknowable in advance and the cost of doing it right is one line.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALAny step that learns parameters from data is part of the model and must be fitted on training data only; this holds for every model family and every modality.
- DATA-SPECIFICThe size of the leak depends on the data: a scaler on a hundred thousand rows leaks almost nothing measurable, while supervised feature selection on a few hundred rows with thousands of candidates can produce an entirely spurious model. Small-n, wide-p data is where preprocessing leakage becomes the whole story.
- SIMULATEDThe observation that global normalisation barely moves the validation/future gap comes from the Leakage Simulator at /ml/leakage, a linear model on synthetic data, and is quoted for the shape of the argument, not as a measurement of the effect on any real system.
Where the depth lives
This domain teaches the model and hands the rest off by name.