Data Leakage
Leakage is information from the answer reaching the model during training through a route that will not exist at prediction time. The offline metric improves; the product does not.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Validation is excellent and production is mediocre, with no skew, no drift and no bug in the serving path. How does information from the label get into the features, and why does the evaluation not notice?
A subscription business wants to know which customers will cancel next month so a retention team can call them. The data science team reports a model that ranks churners almost perfectly on a held-out set. After a month of calls the retention team says the list is no better than calling customers at random.
Gather every column that could plausibly relate to cancelling, join them all onto the subscriber-month table, shuffle, split, train, and read the validation number. More columns cannot hurt: the model will learn to ignore the useless ones.
A column joined from billing is set when an account is cancelled. It is the label with a different name and a little noise. Validation, drawn from the same table, contains it too, so the number is superb and means nothing.
- A column joined from billing is set when an account is cancelled. It is the label with a different name and a little noise. Validation, drawn from the same table, contains it too, so the number is superb and means nothing.
- An activity column was computed over the whole event log, including the weeks after the snapshot. Customers who churned stopped logging in, so "days since last login" knows the future. In production it is computed from the past only and the relationship vanishes.
- The same person's four snapshots are scattered across train and validation, so the model is graded on people it has already memorised. Next month's list is full of people it has never seen.
- Every one of these made the validation metric higher, not lower. The evaluation was correct about the table it was given; the table was not what production will see.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether an active subscriber at the end of month
mwill have cancelled by the end of monthm+1. The label is derived from the billing system's cancellation record. - The decision is a call list of fixed size, so what matters is ranking quality near the top of the list, not a global accuracy.
- One example is one subscriber at one monthly snapshot: account age, plan, usage counts, support tickets, and columns joined from billing and product analytics.
- The snapshot table was assembled by joining today's state of every source system onto a list of subscribers and month-ends. Nobody recorded when each joined column was last updated.
- Several subscribers appear in four consecutive snapshots. The split was a random shuffle of rows.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A supervised model finds whatever function of the features best predicts the label on the training rows. If any feature carries information that was derived from the label, or from events that happened after the prediction time, the cheapest function is to read that feature — and gradient descent or tree splitting will find it before it finds anything real.
- Evaluation only detects this if the validation set lacks the leaked information. When validation is drawn from the same contaminated table by the same split, it contains the same shortcut, and it rewards the model for taking it. The metric is a measurement of the shortcut.
- Production is the first dataset that was built the way the prediction will really be made: as of a moment, from the past only, for entities the model has not seen. That is why the gap appears there and nowhere earlier.
- The leak's size depends on the data. A near-duplicate of the label produces a near-perfect score; a subtle temporal window produces a few points; global normalisation on a linear model produces almost nothing. The reason to refuse all of them is not that every leak is large but that you cannot know its size from inside the contaminated evaluation.
The four injections, and what each does to the number
The simulator at /ml/leakage trains the same logistic regression on the same synthetic subscription data under five conditions. The honest baseline splits by user and standardises with training statistics only; its validation AUC and its AUC on a later month of unseen users agree to within sampling noise. Each injection is then added the way it happens in real pipelines — as a feature or as a split — and both numbers are recomputed.
Illustratively, with the honest baseline near AUC 0.70 on both sets: a target-derived cancellation_flag pushes validation above 0.95 while the future month stays where it was; a days_since_last_login computed over the full event log lifts validation by roughly ten points and the future month by none; scattering each user's four snapshots across train and validation with cohort features opens a gap of several points; global normalisation moves the gap by a fraction of a point. In every case validation rises and the future month does not, because the future month is built the way production is.
The fourth injection is the important teaching case. It is a genuine contamination and its measurable cost here is tiny, because standardising is an affine map and a linear model is nearly indifferent to it. The reason to refuse it anyway is that the size of a leak is a property of the data and the model, not of the mistake, and you cannot measure it from inside the contaminated evaluation.
Validation AUC well above the previous model, on a random row split of the joined snapshot table.
The retention team reports that the top of the list converts no better than a random sample of subscribers; next month's cancellations show the ranking is close to chance among unseen users.
- 1A column joined from billing is set by the cancellation process itself, so it encodes the label; at serving time it is empty for the users being scored and the model's strongest feature is a constant.
- 2An activity aggregate was computed over the whole event log including the weeks after each snapshot, so it knew who had stopped logging in; the serving path computes it from the past only.
- 3Users appear in several snapshots across train and validation, so validation graded the model on memorised people; production users are new to it.
| Injection | Enters as | Validation AUC | Future-period AUC | Lesson |
|---|---|---|---|---|
| None (group split by user) | the honest baseline | agrees with future | agrees with validation | Data Leakage |
| Future timestamp | a feature computed over events after the snapshot | rises sharply | unchanged | Temporal Leakage |
| Target-derived feature | a billing flag written after the outcome | near perfect | unchanged | Target Leakage |
| Same user in train and validation | a random row split plus cohort features | rises | unchanged | Entity Leakage |
| Global normalisation | scaler fitted on all rows before the split | barely moves | barely moves | Preprocessing Leakage |
Leakage is about when, not which
The tempting rule is a blocklist: never use billing columns, never use anything with "cancel" in the name. It is wrong in both directions. A billing column such as the plan a customer was on at the snapshot is known before the prediction and is one of the best features available. And a column with an innocent name — an average computed over "the last 30 days" by a job that ran a week after the snapshot — is leaky.
The only rule that works is a timestamp: for each training row, when would the prediction have been made, and was this value computable from information that existed then? A feature that passes is fine regardless of its source; a feature that fails is leaky regardless of its name.
looks like An integer per subscriber-month, present in training and available from the ticketing system at serving time. Nothing about it looks suspicious.
why it leaks The training table was built by a job that ran on the 7th of the following month and counted tickets in the 30 days before *the job ran*, not before the snapshot. A week of post-snapshot tickets — including the "how do I cancel" ticket — is in the count.
fix Compute the window relative to the row's own prediction timestamp, ending at or before it, from an event table with event timestamps — a point-in-time join, not a join to current state.
The assumption that has to hold after shipping
A leak-free training table is a claim about every feature's timestamp. That claim has to be re-established every time the feature pipeline changes, because the easiest way to introduce a leak is a well-meaning refactor that switches a windowed aggregate from an event table to a current-state table for speed.
So the point-in-time property should be tested, not remembered. A test that rebuilds a sample of rows with an artificially early clock and asserts the features do not change is cheap, and it is the only defence that survives staff turnover.
Every feature value for a training row was computed from information that existed at that row's prediction timestamp, and would take the same value if recomputed with a clock stopped at that timestamp.
holds when Features are built by point-in-time joins against event tables with reliable event timestamps, and the split is by entity or time so no row shares a leaked value with its validation counterpart.
breaks when A feature is joined from a current-state table; a window is computed relative to job run time rather than snapshot time; a source system backfills or updates history in place; a refactor changes the join without changing the feature name.
respond Do not retrain. Find the feature whose value depends on post-snapshot information, rebuild it as-of, re-split, and only then compare the honest number to the baseline the business actually needs to beat.
1def assert_point_in_time(build_features, sample_rows):2 """Rebuild each row's features with the event log truncated at the3 row's prediction time. If the values change, something after the4 snapshot was used."""5 for row in sample_rows:6 full = build_features(row, events=all_events)7 truncated = build_features(8 row, events=all_events[all_events.ts <= row.prediction_ts]9 )10 diff = {k for k in full if full[k] != truncated.get(k)}11 assert not diff, f"row {row.id}: post-snapshot data in {sorted(diff)}"The test is only as good as the event timestamps. If the event table carries the time a record was *loaded* rather than the time the thing *happened*, a truncation at prediction time will keep late-loaded events that happened earlier and drop events that happened before the snapshot but loaded after — the same problem this lesson is about, one layer down.
How to build it
Most important first.
- Build the training table the way the prediction will be made: for each example, fix the prediction timestamp, and compute every feature from data that existed at that moment (Point-in-Time Correctness). This one discipline removes target and temporal leakage together.
- Split before anything is fitted, split by the unit the model will meet fresh in production — entity or time, not row (Group Split, Time-Based Split) — and fit every preprocessing step on the training fold only.
- Hold out a slice that resembles production as closely as possible: a later period, of entities absent from training. Report that number next to the validation number; a gap between them is the leak detector.
- Audit every feature before the model is trained, not after the metric looks suspicious (The Leakage Audit). The cheapest time to find a leak is before anyone has an offline number to defend.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The gap between validation AUC and the AUC on a later period of unseen entities. Validation alone cannot detect leakage because it shares the leak; the future-period number is the one that resembles production.
- The delayed production ranking quality once next month's cancellations are known — the only number the retention team cares about, and the last one available.
- A validation number on its own, however high, is not evidence of anything until the split and the feature timestamps have been inspected.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every feature value in the training table was computable from information available at that row's prediction timestamp, and the serving path computes it from the same information.
- No entity that appears in the validation or test set appears in the training set, unless the model will genuinely be scoring entities it has seen before at the same rate in production.
- Every preprocessing statistic, encoder and selected feature was derived from the training fold alone.
- Offline: retrain with the single most important feature removed. If the metric collapses toward the baseline, that feature is either the whole model or a leak; inspect its computation and timestamp before believing either.
- Offline: score a later period of unseen entities with features rebuilt as-of the snapshot. Compare against validation; a gap larger than the sampling noise of either number is a leak until proven otherwise.
- Online: when next month's outcomes arrive, compute ranking quality on the served list and compare it to the future-period number, not to validation.
What can go wrong
- The future-period holdout is built from the same contaminated snapshot table, so it carries the same leaked columns and agrees with validation. Both numbers are wrong together.
- The leak is removed, the honest number is much lower, and the project is cancelled as "the model does not work" — when the honest number was always what production would deliver and may still beat the current call list.
- A feature is fixed to be point-in-time correct in training but the serving path still reads the live column, so the leak becomes skew (Train / Serve Skew).
- Point-in-time feature construction is slower and harder to write than joining current state, and needs history tables the source systems may not keep.
- An honest holdout of later, unseen entities is smaller than a random validation set, so its confidence interval is wider (Metric Uncertainty).
- Removing the leak lowers the reported number, which is politically expensive when a previous number has already been promised.
- "The model overfit; add regularisation." Regularisation cannot remove a feature that is legitimately the best predictor of the label on the training table. The problem is in the table, not the fit.
- "Production data must have drifted." Drift is a change over time; leakage is a gap present on day one. A day-one gap with stable feature distributions points at leakage or skew, not drift.
- "Drop every column that comes from billing." Some billing columns are known before the prediction time and are valuable. The rule is about when a value was computed, not which system it came from.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a model will exploit any feature carrying label information follows from what supervised learning optimises, so the mechanism is the same for a linear model, a tree ensemble or a network, on any modality.
- SIMULATEDThe AUC figures quoted for the four injections come from the Leakage Simulator at /ml/leakage — a logistic regression trained on a seeded synthetic subscription dataset. They show the shape of each leak, not a measurement on any real system, and they move with the seed.
- DATA-SPECIFICThe size of a leak depends on the data: a target-derived column produces a near-perfect validation score, entity overlap with per-entity features produces a large gap, and global normalisation on a linear model produces almost no measurable gap at all. The discipline is the same whether the leak is large or small.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — the stopped-clock test is a property test over the feature pipeline, and keeping it green when the pipeline legitimately changes is a testing discipline this domain assumes.