FormulationGENERALSIMULATED

Label Leakage

A feature that carries the answer — cancelled_at used to predict will_cancel — gives excellent offline metrics and an invalid model. Leakage is about when information exists, not which columns are forbidden.

Target & data What to measure What must stay true

The problem, the obvious approach, and why it breaks

Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.

The question

How does information from the label reach the features, why does the offline evaluation reward it, and how do you tell a leaked feature from a legitimately predictive one?

The problem

The data scientist, delighted: "The new churn model is nearly perfect on validation. The top feature is days-since-last-billing-event. Can we ship it this week?"

The obvious approach

Drop the obviously bad columns — cancelled_at, churned — and use everything else. If a feature is strongly predictive, that is what we want; the model found signal.

Why it breaks

The feature was computed from the end state, so for churners it encodes how long ago their last billing event was *as of the build date* — which, for a subscriber who cancelled, is long ago, and for one who stayed, is recent. It is the label, transformed. Validation rewards it because validation rows carry the same end state.

How it breaks — usually after the offline metric looked fine

The feature was computed from the end state, so for churners it encodes how long ago their last billing event was *as of the build date* — which, for a subscriber who cancelled, is long ago, and for one who stayed, is recent. It is the label, transformed. Validation rewards it because validation rows carry the same end state.
At the prediction moment the subscriber has not cancelled yet; their last billing event is the most recent monthly charge. The feature takes the "loyal" value for everyone, and the model, which leaned on it, predicts loyalty for everyone.
Dropping the named label columns caught nothing, because the leak is not in a column name. It is in the *time* the column was computed.
The evaluation number that justified the launch was correct — for a system that knows the future. There is no such system in production.

Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

What is being predicted, and from what data

This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.

Target

Whether paid access ends within thirty days of the monthly snapshot, as constructed in Label Construction. The label is a function of events strictly after the snapshot.
The lesson's target is the boundary: every feature must be a function of events at or before the snapshot, and the evaluation must be unable to reward one that is not.

Data

The feature table was built by joining the monthly snapshot to the current subscriptions table and the events table without a time filter. days_since_last_billing_event was computed from the most recent billing event on record — including the final one that marks the end of the subscription.
The validation set is a random sample of (subscriber, snapshot) rows from the same feature table.

How it actually works

Precisely enough to predict its behaviour — not a framework API.

Label leakage is any path by which information that is only available after the prediction moment — the label itself, an event that follows from it, or a value updated because of it — reaches a feature. The model learns the path because it is the strongest signal in the table, and the offline evaluation rewards it because the validation rows were built the same way and carry the same future.
The common paths: a mutable state column read at build time rather than at the snapshot (plan, status, last_event); an aggregate computed over a window that extends past the snapshot; a feature updated *because of* the outcome (a support ticket tagged "cancellation", a refund); and the label itself under another name (Target Leakage, Temporal Leakage).
The distinguishing test is not "is this column related to churn" — good features are — but "would this value have been known, with this value, at the snapshot moment". A point-in-time join, which reconstructs each feature as of the snapshot, is the mechanism that enforces it (Point-in-Time Correctness).

The feature that was the label

The canonical case is cancelled_at used to predict will_cancel, and nobody does that by name. They do it by computing a feature from a table whose state was updated by the cancellation. days_since_last_billing_event is that feature: for a subscriber who cancelled, the last billing event is the final one, months before the build date; for one who stayed, it is last month. The feature is a clock that started when the label happened.

Everything about the offline evaluation confirms it. The feature dominates importance; the validation number is superb; the calibration is perfect. Every one of those is a measurement of how well the model reads the future from a table that contains it.

leakagedays_since_last_billing_eventThe end state, read at build time

looks like A sensible recency feature — how long since the subscriber was last charged — with a strong, monotonic relationship to churn and a plausible story.

why it leaks It was computed from the current events table without a time filter, so its value reflects the subscriber's *final* billing event. For churners that is the last charge before they left; the feature encodes "has stopped being billed", which is the label.

offline

Validation rows were built the same way and carry the same end state, so the model's reliance on the feature is rewarded: near-perfect ranking, dominant importance, and a number that looks like a breakthrough.

production

At the snapshot, every active subscriber was billed within the last month, so the feature takes the "loyal" value for everyone and the model predicts no churn for anyone. The retention list is empty.

fix Compute the feature with a point-in-time join — billing events with time ≤ snapshot — and validate on a period after the training window, where a future-reading feature cannot be rewarded.

when this feature is fine As of the snapshot, days since the last billing event is a perfectly legitimate feature: a subscriber whose last charge failed and was not retried is genuinely more likely to lapse, and that is known at prediction time.

Why validation cannot see it

The offline evaluation is not lying. It is measuring exactly what it was given: a table where every row, training and validation alike, was built at the same moment from the same future-containing state. Holding out rows does not hold out the future. The only split that can catch this is one where the validation rows come from a period the feature computation could not have seen.

The gap between the two numbers below is the size of the leak. The cost of seeing it is a smaller validation set and a period of data withheld from training — and a much smaller number to present.

The recency-feature churn model

offline evaluation said

Near-perfect ranking on a random hold-out from the feature table; days_since_last_billing_event at the top of every importance chart.

production did

Deployed to shadow for one snapshot: the model ranked every active subscriber as low risk, the retention list was empty, and the future-period evaluation on the same artifact was barely above the majority baseline.

What explains the gap — most likely first

1The feature was computed from end state at build time, so it encoded the label; at serving time the end state does not exist yet and the feature is uninformative.
2The random split placed rows with the leaked value on both sides, so validation rewarded the leak instead of exposing it.
3The remaining features carry modest, honest signal that the model under-weighted because the leak explained the label on its own.

what it costs to close or detect Detecting it required a future-period evaluation, which meant withholding the most recent months from training and accepting a noisier number; fixing it required an events table with a point-in-time join, which Data Engineering had to build, and a model whose honest number was a fraction of the one already presented to leadership.

The boundary as code

The rule that prevents this is a time filter, and it belongs in the feature computation, not in a review checklist. Every feature for (subscriber, t) is a function of events with time ≤ t. The query below computes the recency feature correctly; the difference from the leaked version is one predicate.

The same predicate is what the serving path applies implicitly — it can only see what exists at request time — which is why the honest training feature and the serving feature agree, and the leaked one and the serving feature do not.

Point-in-time recency — the predicate that removes the leak

1-- Feature for (subscriber, snapshot_date): days since last billing event
2-- *as of the snapshot*. The WHERE clause is the entire difference.
3SELECT sn.subscriber_id,
4       sn.snapshot_date,
5       sn.snapshot_date - MAX(b.event_time)::date AS days_since_last_billing_event
6FROM   snapshots sn
7LEFT JOIN billing_events b
8  ON   b.subscriber_id = sn.subscriber_id
9 AND   b.event_time   <= sn.snapshot_date        -- nothing from after the moment
10GROUP  BY sn.subscriber_id, sn.snapshot_date;
11 
12-- The leaked version had no time predicate: MAX(b.event_time) was the
13-- subscriber's final billing event, whenever it happened.

Test it by replay: pick a snapshot from six months ago, compute the feature with and without the predicate, and diff. For churners the two values differ by the length of time since they left; for everyone else they are equal. That diff is the leak, measured.

How to build it

Most important first.

Build every feature with a point-in-time join: the feature for (subscriber, t) is computed only from events with time ≤ t, using the events table rather than current state (Point-in-Time Correctness, Temporal Features).
Validate on a period strictly after the training period, so that a feature that leaks the future cannot be rewarded by validation rows that share it (Time-Based Split).
Audit the top features of every model: for each, name the source, the computation window and the snapshot alignment, and be suspicious of any feature whose importance is out of proportion to its plausibility (The Leakage Audit).
Replay: compute the feature vector for a sample of last month's snapshots using only data that existed at the time, and diff against the training-time vector; any difference is a leak or a skew (Serving Contract Tests).

What to measure

Which number actually maps to the decision — and which numbers look relevant and are not.

The gap between validation quality on a random split and on a future-period split. A large gap with the same features is the signature of leakage; the future-period number is the one that describes production.
Per-feature: the difference between its value as computed at build time and as computed with a point-in-time filter, over a sample of rows. Nonzero is a leak.
Do not measure "top feature importance" as a sign of a good model. The most important feature in a leaked model is the leak.

What must stay true after deployment

The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.

Assumptions

Every feature the model reads is still computed from events at or before the snapshot, and no upstream table it reads has become mutable or started carrying post-snapshot values.
The serving-time feature computation still has the same time boundary as the training-time one — the replay diff is still zero.
No process writes information derived from the outcome, or from the model's own prediction, into a source the features are computed from.

How to verify — offline, online, and over time

Offline: the leakage audit for every top feature; a future-period evaluation alongside the random one; a replay diff of point-in-time features against training features on a sample.
Online: a test at serving that every feature's source timestamp is ≤ the request time; a monitor on features whose serving distribution differs sharply from training — a leaked feature collapses to one value in production.
Over time: re-run the audit whenever a source table, a feature definition or a downstream write path changes; a sudden improvement in offline metrics after a pipeline change is a leak until proven otherwise.

What can go wrong

Failure modes in production

The point-in-time join is implemented and the snapshot timestamp is the wrong one — the end of the day rather than the moment of the decision — so features include the evening's events for a morning prediction.
The future-period validation is honest but the features were selected on the random split first; the leaked features were dropped, but the choice of which to keep was informed by the leak (Evaluation Leakage).
A downstream system writes a value back into the source table when a subscriber is contacted by retention, and a "legitimate" feature — support contact count — starts carrying the model's own predictions (Feedback Loops).

What the recommended approach costs

Point-in-time joins over an events table are more expensive than a join to current state, and Data Engineering has to maintain the event history to make them possible.
Future-period validation uses less data and produces a smaller, noisier number than a random split; the honest number is harder to sell.
Removing the leaked feature removes most of the offline signal, and the honest model looks like a step backwards to anyone who saw the leaked number first.

Misreads

"We dropped the label column, so there is no leakage." Leakage is about when a value was computed, not what it is called. Any column read from mutable state at build time can carry the label.
"The feature is genuinely predictive, so it is not a leak." A leaked feature is the most genuinely predictive one in the table — offline. The question is whether it would have had that value at the prediction moment.
"Validation was held out, so the number is honest." A random hold-out from a table with leaked features holds out rows, not the future. The validation rows carry the leak too.

Where this applies

ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

GENERALThe mechanism — post-moment information reaching a feature, rewarded by validation rows that share it — is independent of model family, task and domain; only the specific paths differ between a churn table and a medical record.
SIMULATEDThe leakage simulator in the Data Leakage module injects each leak into a synthetic dataset and compares random-split validation against a future-period evaluation; its numbers are produced by that model and illustrate the gap, not any real system.

Where the depth lives

This domain teaches the model and hands the rest off by name.

Data Engineeringscd-type-2 snapshot-tables keeping-raw-history event-time