Target Leakage
A feature that is derived from, caused by, or written by the same process as the label. It looks like a column; it is the answer.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A feature predicts the label almost perfectly on the training table. How do you tell whether you have found a strong signal or a copy of the answer?
A payments team wants to predict which transactions will be disputed so they can hold them for review before settlement. Their first model separates disputed from undisputed transactions almost perfectly on validation. When it is put in the review path it holds almost nothing, and the few holds it makes look random.
Join every column from the operational database onto the transactions, including the ones about disputes, because a model should be given all available information and will learn which parts matter.
chargeback_flag is set by the process that produces the label. It is the label. The model learns to read it, validation is near perfect, and every other feature receives almost no weight.
chargeback_flagis set by the process that produces the label. It is the label. The model learns to read it, validation is near perfect, and every other feature receives almost no weight.- At authorisation time no transaction has a chargeback flag, a refund or a dispute timestamp — those all arrive later. The serving path fills them with nulls or the training default, the model's strongest features are constants, and the remaining features were never learned properly because the flag was always available to lean on.
- Subtler columns carry the same information indirectly:
refund_issuedis often the merchant's response to a dispute; arisk_review_idis assigned only to transactions that were flagged, which happens after the outcome;days_to_settlementis longer for transactions that were held. Each is a proxy that would not exist at prediction time.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict at authorisation time whether a card transaction will result in a chargeback within 90 days. The label comes from the issuer's dispute feed, which arrives weeks to months later.
- The decision — hold or settle — must be made before settlement, so only information available at authorisation can be used.
- One example is one transaction with card, merchant and amount features, joined to the payments team's operational database, which holds a
chargeback_flag, arefund_issuedcolumn and adispute_opened_attimestamp for transactions that were disputed. - The operational table is the current state of each transaction. Columns are overwritten as the transaction's life proceeds; there is no history of what each column held on the authorisation date.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Target leakage is a causal direction error. The feature does not predict the label; the label, or the process that generates the label, produces the feature. On the training table the two are correlated because they were both written after the outcome; at prediction time the outcome has not happened, so the feature has no value or a default.
- It is invisible to any evaluation on the same table because the validation rows were assembled the same way. Cross-validation, a larger holdout and a different metric all report the same near-perfect number, because they all read the same column.
- The proxy form is harder to spot: a column that is not the label but is only ever set *because of* the label. An id assigned after approval, a status updated after a dispute, a note added by the fraud team. The name gives nothing away; the mechanism that writes the column does.
- Current-state tables make this the default outcome. When a row is overwritten as the entity's life proceeds, a join to that table today returns values from the end of the story, not from the moment the prediction would have been made.
The label under another name
The operational table holds one row per transaction and overwrites it as the transaction's life proceeds. When a dispute arrives, chargeback_flag flips to true, dispute_opened_at is filled in and, often, refund_issued is set when the merchant concedes. Joining that table onto a training set today returns the end of every transaction's story.
The model does not know it is reading the future. It sees a binary column that agrees with the label on nearly every row and puts nearly all of its weight on it. Every other feature is learned only to the extent it explains the small residual noise where the flag and the label disagree.
looks like A boolean column in the payments database, present on every transaction row, cheap to join, and near-perfectly correlated with the dispute label.
why it leaks The flag is set by the dispute-handling process, which runs after the issuer reports a chargeback — the same event the label is derived from. It is not a predictor of the outcome; it is a record of it.
fix Exclude every column written by or after the dispute process. Rebuild the remaining features as of authorisation time from event history, and confirm by logging serving-time availability per feature.
Proxies: columns that exist because of the outcome
Once the obvious flag is gone, the remaining leaks are columns nobody would name as the label. A review id that is assigned only when a transaction is flagged. A settlement delay that is longer because the transaction was held. A merchant risk tier the fraud team raises after disputes accumulate. Each is a downstream consequence of the outcome that happens to be stored as an attribute.
The test is not the column name or its source system but the process that writes it. If the writing process runs after the prediction time, or runs only because of the outcome, the column is a proxy. The feature availability table below is the cheapest way to surface the ones the eye misses.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
risk_review_id joined as a feature | Non-null almost only on positive rows; very high importance | Assigned when a transaction enters review, which happens after a dispute or a hold | Exclude; the fact of a review is a consequence of the outcome, not information available at authorisation |
days_to_settlement as a feature | Strongly right-skewed for positives | Held transactions settle late; holds are caused by disputes or by a previous model's decision | Exclude; also a feedback loop with the current review policy |
merchant_risk_tier from a static-looking dimension table | Looks like a fixed attribute; strong feature | The tier is updated in place when a merchant accumulates chargebacks, so today's tier encodes disputes that came after many training transactions | Rebuild as of the transaction date from the tier change history, or exclude if no history exists |
refund_issued as a feature | Near-perfect on the subset where it is set | Merchants refund in response to a dispute; the column is written by the outcome | Exclude; refunds that happened *before* the transaction on the same card are a different, legitimate feature |
Serving availability is the detector
A leaked feature has a signature that no offline evaluation shows but the serving path reveals on day one: it is available in training and absent in serving. Logging the fraction of requests in which each feature takes a real value, and comparing it to the training table, catches target leakage without waiting weeks for chargeback labels.
This is also why the ablation matters. A model that loses most of its quality when one feature is removed is either a single-feature model, which should be a rule, or a leaked model, which should be rebuilt. Either way the number as reported should not be believed until the ablation has been run.
Each feature is available at serving time with a real value at the same rate as in the training table.
holds when Features are computed from event history as of the prediction time in both paths, and no column comes from a process that runs after the outcome.
breaks when A current-state table is joined directly; a re-score happens late enough that outcome-derived columns have been filled for some requests; a source system starts backfilling a column.
respond Remove the feature, rebuild the table as-of, retrain, and report the honest number against the business baseline rather than against the leaked one.
1def leak_suspects(train_df, serving_log, label, model_metric, retrain):2 suspects = []3 for f in train_df.columns.drop(label):4 avail_train = train_df[f].notna().mean()5 avail_serve = serving_log[f].notna().mean()6 # strong in training, missing in serving: was never known at prediction time7 if avail_train > 0.9 and avail_serve < 0.5:8 suspects.append((f, "unavailable at serving"))9 # non-null almost only on positives: written because of the outcome10 pos_rate = train_df.loc[train_df[f].notna(), label].mean()11 if pos_rate > 5 * train_df[label].mean():12 suspects.append((f, "present mostly on positives"))13 # ablation: does the model survive losing its top feature?14 top = model_metric.top_feature()15 if retrain(train_df.drop(columns=[top])).metric < 0.5 * model_metric.value:16 suspects.append((top, "model collapses without it"))17 return suspectsThe availability check needs a serving log, which means a shadow deployment or at least a dry run of the feature path on live requests before the model is promoted — the cost of catching target leakage early is that serving-side logging has to exist before there is a model worth serving.
How to build it
Most important first.
- For each candidate feature, name the process that writes it and ask whether that process runs before or after the prediction time. Anything written by, after, or because of the outcome process is excluded — regardless of how predictive it looks.
- Build features from event history with timestamps and take the value as of the prediction time (Point-in-Time Correctness), never from a current-state table. If a source only keeps current state, the feature is unavailable for training, however useful it would be.
- Treat a near-perfect validation number as a bug report. Run the ablation: remove the top feature and retrain; if the metric collapses, that feature is either a leak or the whole model, and either way it needs a written justification.
- Confirm with the serving path: for a sample of production requests, log what value each feature actually takes at request time. A feature that is always null or always the default in production was never available.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The per-feature availability rate at serving time — the fraction of requests in which the feature has a real value rather than a null or default. A strong training feature with near-zero serving availability is target leakage until shown otherwise.
- The gap between validation on the joined table and a holdout rebuilt from authorisation-time information only.
- Not the validation number, which measures how well the model reads the leaked column.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- No feature in the training table is written by, after, or as a consequence of the process that produces the label.
- For every feature, the serving path supplies a real value at the same rate and from the same information as the training table did — a feature available in training but null in serving is a leak that has become skew.
- Source systems that overwrite rows are not joined directly; every feature comes from a timestamped event history read as of the prediction time.
- Offline: for each feature, compute the correlation with the label and the fraction of rows where it is non-null; a feature that is both very strong and mostly null — or non-null only for positives — is the signature of a post-outcome column.
- Offline: ablate the top feature and retrain. A collapse toward the baseline is a leak or a single-feature model; both need investigation before the number is reported.
- Online: log the serving-time feature vector and compare per-feature availability against training. Any feature whose serving availability is far below its training availability was not known at prediction time.
What can go wrong
- The obvious columns are removed and a proxy survives: a
merchant_risk_tierthat the fraud team updates *after* a merchant accumulates disputes, so it encodes past labels for merchants but reads as a static attribute. - The leak is removed from training, but the serving path still joins the live operational table, and for the small fraction of transactions that were already disputed by the time of a re-score the flag is present — producing a model that is confidently right only on transactions that no longer need a prediction.
- The honest model is much weaker than the leaked one, and the team keeps the leaked column "because it is available for some transactions", building a model whose quality depends on how late the prediction is made.
- Excluding every post-outcome column removes the strongest-looking features and produces a much lower reported number — the honest number, but a harder one to defend.
- Point-in-time feature construction needs event history that operational systems often do not keep, so some genuinely legitimate features become unavailable for training until the history is captured.
- Per-feature availability logging in serving is extra storage and a schema to maintain.
- "It is a strong feature; we should keep it and just make serving supply it." Serving cannot supply the result of a process that has not run yet. A feature that is only available after the outcome is available only when the prediction is no longer needed.
- "Remove the column and the leak is gone." Removing
chargeback_flagleavesrefund_issued,dispute_opened_at,risk_review_idand any column that is only ever set because of a dispute. The check is per feature, by the process that writes it. - "Feature importance shows this column matters, so it is a real signal." Importance measures what the model used, not whether the value existed at prediction time. A leak is always important.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALA feature written by the label-generating process leaks under any model family and any task; the mechanism is about the data, not the learner.
- DOMAIN-SPECIFICProxy leaks take the domain's shape: in payments it is refund and review columns, in healthcare it is a treatment or billing code that is only recorded after a diagnosis, in hiring it is an interview stage reached only by candidates who were later accepted. The question — what writes this column, and when — is the same.
Where the depth lives
This domain teaches the model and hands the rest off by name.