LeakageGENERALDATA-SPECIFICCONTESTED

Temporal Leakage

Information from after the prediction time reaches the features: a future timestamp, a window that crosses the snapshot, a random split of time-ordered data.

Target & data What to measure What must stay true ▶ Leakage Simulator

The problem, the obvious approach, and why it breaks

Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.

The question

The features are honest columns and none of them is the label. How can they still know the future, and why does a random split hide it?

The problem

A retailer wants a weekly forecast of which products will go out of stock so they can reorder early. The model validated well and was rolled out; the buyers say its warnings arrive after the shelf is already empty, and that on paper it "predicted" stockouts that had already happened.

The obvious approach

Compute rolling windows for every product-week, shuffle, split 80/20, validate. The windows are the same feature definitions used in serving, so there is no skew, and a random split gives a large validation set.

Why it breaks

The training job's "last 7 days" window ends on Sunday night, after the week being predicted. It contains the week's own sales — including the days the product was sold out. The feature knows the outcome.

How it breaks — usually after the offline metric looked fine

The training job's "last 7 days" window ends on Sunday night, after the week being predicted. It contains the week's own sales — including the days the product was sold out. The feature knows the outcome.
The random split puts week 30 in validation and weeks 29 and 31 in training. Sales for a product are autocorrelated week to week, so the model interpolates from neighbouring weeks it has seen, which it cannot do in production where every week is the newest.
A last_restock_date feature was joined from the current inventory table, so for training rows it holds the restock that happened *after* the stockout — a perfect signal that a stockout occurred.
Validation was strong on all counts. In production, every window ends at Monday morning, every week is the future, and the restock date is the previous one. The warnings are late because the model never learned to warn early; it learned to recognise a week that already contained a stockout.

Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

What is being predicted, and from what data

This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.

Target

Predict, as of Monday morning, whether a product at a store will stock out before the following Monday. The label is the first zero-inventory scan in that week.
The prediction is only useful if it is made with what was known on Monday morning; a prediction that uses Wednesday's sales is a report, not a forecast.

Data

One example is one product-store-week with rolling sales, inventory and promotion features, computed by a batch job over the sales event table.
The batch job computes "sales in the last 7 days" relative to the day the job runs, and it runs on Sunday night for the *preceding* week when building the training set, but on Monday morning for the *current* week in serving.
Training rows were shuffled and split at random across all weeks of the year.

How it actually works

Precisely enough to predict its behaviour — not a framework API.

Temporal leakage is any path by which the feature value for a row depends on events with a timestamp later than that row's prediction time. It enters through windows computed relative to the wrong clock, through joins to current state, and through a split that lets the model see rows from the future of the ones it is scored on.
The window case is the commonest and the subtlest. A window is defined by its end. "Last 7 days" relative to job time, snapshot time, event time and label time are four different features; only the one ending at or before the prediction time is legitimate (Temporal Features).
The random-split case leaks differently: no feature is wrong, but the model is evaluated on a period whose neighbours it has trained on. For anything with autocorrelation — sales, prices, sensor readings, user activity — that makes validation an interpolation task, while production is always extrapolation (Random Split, Time-Based Split).
The simulator's future-timestamp injection is the window case in miniature: days_since_last_login computed over the whole event log is large exactly for users who churned, because they stopped logging in *after* the snapshot.

A window is defined by its end

The feature was called sales_last_7d in both the training job and the serving service, and both computed a seven-day sum. The training job ran on Sunday night and summed the seven days before it ran — the week being predicted. The serving job ran on Monday morning and summed the seven days before *it* ran — the previous week. Same name, same length, different end, and only one of them is a forecast feature.

The fix is to make the window end an explicit argument that comes from the row, not from the clock. Once every row carries a prediction_ts, the window is [prediction_ts - 7d, prediction_ts) in both paths and cannot silently drift with the job schedule.

leakagesales_last_7dA window that crosses the snapshot

looks like A rolling sum with the same name and definition in the training SQL and the serving code. Nothing about it looks temporal.

why it leaks In training the window is anchored to the job's run time, which is after the week being predicted, so the sum includes the week's own sales — and the zero-sales days after the stockout.

offline

The feature is among the strongest in the model, and validation, whose rows were built by the same job, confirms it.

production

The serving window ends at Monday morning and contains only the prior week. The feature's relationship to the label is now the honest, weaker one, and the model's warnings arrive late because it learned to recognise a stockout week, not to anticipate one.

fix Anchor the window to the row's prediction timestamp in both paths: [prediction_ts - 7d, prediction_ts). Add a test asserting window_end <= prediction_ts for every windowed feature.

when this feature is fine The same seven-day sum ending at Monday morning is exactly right: the previous week's sales are known when the forecast is made, are computed identically in serving, and are a genuine predictor of demand.

Point-in-time window, anchored to the row

1-- one row per (product, store, prediction_ts); window ends AT the prediction time
2SELECT r.product_id, r.store_id, r.prediction_ts,
3       COALESCE(SUM(s.qty), 0) AS sales_last_7d
4FROM   prediction_rows r
5LEFT JOIN sales_events s
6  ON   s.product_id = r.product_id
7  AND  s.store_id   = r.store_id
8  AND  s.event_ts  >= r.prediction_ts - INTERVAL '7 days'
9  AND  s.event_ts  <  r.prediction_ts          -- never at or after
10GROUP BY 1, 2, 3;

The strict < on prediction_ts is the whole lesson. The join is against an event table with event timestamps, not a daily aggregate table whose rows may have been rebuilt after the fact.

Random splits interpolate; production extrapolates

Even with every window correct, a random split of product-weeks lets the model see week 29 and week 31 when it is scored on week 30. Sales are autocorrelated, so the model can effectively average its neighbours. In production it is always asked about the newest week with only earlier weeks available.

This is not a feature bug and no audit of columns will find it. It is a mismatch between the evaluation's question and production's question. The remedy is a split that respects time, with a gap so the last training label is not computed from events inside the validation period.

Stockout forecast, first month in production

offline evaluation said

Strong classification metrics on a random 80/20 split of product-weeks across the year.

production did

Buyers report warnings with negative lead time — the shelf was already empty — and no reduction in stockouts; the time-split re-evaluation after the fact shows a much weaker model.

What explains the gap — most likely first

1The training window ended after the predicted week, so the feature contained the outcome; serving windows end at the prediction time and do not.
2The random split let the model interpolate between neighbouring weeks of the same product; production only extrapolates.
3last_restock_date was joined from the current inventory table and encoded the restock that followed each stockout.

what it costs to close or detect An honest evaluation is a single later period after a gap — smaller and noisier than the random split — and needs rolling-origin retrains to become stable. Rebuilding windows as-of requires the event table rather than the convenient daily aggregate, and a history table for restocks that did not exist.

Event time is the clock that matters

A correct window needs a correct clock. If the sales table's timestamp records when a row was loaded into the warehouse rather than when the sale happened, then a window ending at the prediction time still admits late-loaded sales that happened after the snapshot and excludes sales that happened before it but loaded late. The same leak, one layer down.

This is the boundary with data engineering: Event Time versus processing time, Late-Arriving Data and watermarks are their problem to solve, and this domain's job is to insist that the feature pipeline uses event time and to test that it does.

must stay trueWindows end before the prediction

Every windowed feature's window ends at or before the row's prediction timestamp, measured in event time, in both training and serving.

holds when Windows are anchored to a per-row prediction_ts; features are computed from event tables with event timestamps; the split is by time with a gap of at least the label horizon.

breaks when A pipeline change anchors a window to run time; a source switches from event to load timestamps; a daily aggregate table is rebuilt after the fact and used as a source; a backfill rewrites history.

how you would know A data test asserting window_end <= prediction_ts per feature; the stopped-clock rebuild diff; a random-split versus time-split gap; negative lead time in production.

respond Fix the window anchor or the timestamp source, rebuild the training table, re-split by time, and re-report the honest number. Do not retrain on the same table.

How to build it

Most important first.

Give every training row an explicit prediction timestamp and compute every window to end at or before it. Make the end of the window a parameter of the feature, not an accident of when the job ran.
Split by time: train on weeks before a cutoff, validate on the weeks after it, with a gap at least as long as the label horizon so no training label is derived from events inside the validation period (Time-Series Validation).
Join slowly-changing attributes as of the prediction time from a history table, never from current state (Point-in-Time Correctness).
Run the stopped-clock test from Data Leakage: rebuild a sample of rows with the event log truncated at the prediction timestamp and assert nothing changes.

What to measure

Which number actually maps to the decision — and which numbers look relevant and are not.

Metric on a time-based holdout after a gap, against the metric on a random split of the same rows. The difference is the size of the temporal leak plus the size of the interpolation advantage; either alone is enough to distrust the random-split number.
Lead time in production: how many days before the stockout the warning arrived. This is the number buyers care about and the one a leaked model destroys.
A random-split validation number on time-ordered data is not a measurement of forecasting quality at all.

What must stay true after deployment

The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.

Assumptions

Every window in every feature ends at or before the row's prediction timestamp, in training and in serving, and the end is computed from the same clock in both.
Rows the model is evaluated on come from a period strictly after every row it was trained on, separated by at least the label horizon.
Event timestamps in the source tables record when events happened, not when they were loaded, and late-arriving events are handled the same way in training and serving.

How to verify — offline, online, and over time

Offline: compare the random-split and time-split metrics. Then run the stopped-clock rebuild on a sample of rows and diff the feature values; any change is a leak.
Offline: for each windowed feature, assert window_end <= prediction_ts in the feature pipeline tests, as a data test rather than a code review item.
Online: measure lead time — the distance between the warning and the observed stockout — on the first weeks of production; a model that learned from the future produces warnings with negative lead time.

What can go wrong

Failure modes in production

The split is fixed to be time-based but the window bug survives, so validation rows still contain their own week's sales; the time split hides nothing because both sides carry the same leak.
The gap between training and validation is shorter than the label horizon, so the last training week's label was computed from events inside the validation period.
Event timestamps are actually load timestamps (Event Time versus ingestion time), so a window that looks correct still includes late-loaded events from after the snapshot and excludes early events that loaded late.

What the recommended approach costs

A time-based holdout is one period, not a random sample, so its estimate is noisier and may reflect that period's peculiarities; rolling-origin validation costs several retrains to fix that.
Ending windows at the prediction time discards the most recent data from the feature, which for slow pipelines can mean features are hours to days staler than the raw data (Feature Freshness).
Point-in-time joins against history tables are slower and need the history to exist; many source systems keep only current state.

Misreads

"There is no leak; the feature definitions are identical in training and serving." Identical definitions relative to *job time* produce different windows when the jobs run at different times. The definition has to be relative to the prediction timestamp.
"Random split gives more validation data, so it is a better estimate." It is a more precise estimate of the wrong quantity. On autocorrelated data the random-split number measures interpolation, which production never asks for.
"The time split fixed it." The split fixes the evaluation; it does not fix a window that crosses the snapshot. Both are needed.

Where this applies

ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

GENERALAny feature that depends on events after the prediction time leaks regardless of model family; any autocorrelated series makes a random split an interpolation test. Both hold for tabular, time-series and sequence models alike.
DATA-SPECIFICThe interpolation advantage of a random split is large when consecutive rows are strongly autocorrelated — weekly sales, hourly load, a patient's consecutive visits — and negligible when rows are independent draws, such as unrelated one-off transactions, where a random split is legitimate.
CONTESTEDSome practitioners argue that for stationary series with short memory a random split is acceptable and a time split throws away data for little gain, and that the gap requirement is overcautious when labels resolve quickly. That is defensible when stationarity has actually been checked and the horizon is short; the counterargument is that stationarity is an assumption nobody verifies, and the time split is the only one that measures what production will do.

Where the depth lives

This domain teaches the model and hands the rest off by name.

Data Engineeringevent-time late-arriving-data windows sliding-windows

Databasewindow-functions