EvaluationGENERALTASK-SPECIFICCONTESTED

Time-Series Validation

When the model will predict the future, validate on the future: forward-chaining folds, a gap between training end and validation start equal to the label delay, and never a shuffle. A random split on temporal data is a leakage simulator with a nicer name.

Target & data What to measure What must stay true ▶ Leakage Simulator

The problem, the obvious approach, and why it breaks

Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.

The question

A churn model validated with a random split looks excellent and degrades within a month of deployment. What did the split let the model see, and what does an honest temporal validation look like?

The problem

A subscription business trains a churn model on two years of customer history and validates it with a random split. Validation looks excellent. Deployed, it is barely better than the old rule within a month, and the retention team has stopped trusting the scores it produces.

The obvious approach

Shuffle all customer-weeks, hold out twenty percent, validate. The held-out rows are real customers with real outcomes and the model has never seen them.

Why it breaks

The model has seen them. A customer's week twelve is in training and week thirteen is in validation, and the two rows are nearly identical, so the model recognises the customer rather than predicting anything (Entity Leakage).

How it breaks — usually after the offline metric looked fine

The model has seen them. A customer's week twelve is in training and week thirteen is in validation, and the two rows are nearly identical, so the model recognises the customer rather than predicting anything (Entity Leakage).
Training rows from the second year sit beside validation rows from the first. The model learned the pricing change's effect on churn from the future and was validated on the past, where that knowledge is a gift (Temporal Leakage).
The label for a validation week is a cancellation in the thirty days after it; training rows from inside those thirty days carry features — a support contact, a downgrade — that are consequences of the decision to cancel. The model learned the aftermath and called it a predictor.
Production has none of this. Every prediction is about the future, for customers whose recent weeks are all it has, under conditions no training row contains. The validation number described a game the deployed model never gets to play.

Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

What is being predicted, and from what data

This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.

Target

Predict whether a customer will cancel within the next thirty days. The label is a cancellation event observed thirty days after the prediction date — so the label for any prediction made today does not exist for a month.
The decision is which customers to contact with a retention offer, made weekly from the current scores.

Data

One example is one customer-week: usage, billing, support contacts, and tenure up to that week. Each customer contributes a row per week, and adjacent weeks for the same customer are nearly identical.
The world changes over the two years: a pricing change, a new competitor, a product redesign. The relationship between usage and churn is not the same in the second year as in the first.

How it actually works

Precisely enough to predict its behaviour — not a framework API.

A model deployed to predict the future is evaluated honestly only on data from after everything it was trained on. Forward-chaining validation does that repeatedly: train on everything up to time t, validate on the window after t, move t forward, repeat. Each fold is a rehearsal of a deployment, and the sequence of fold metrics shows how performance moves as the world changes.
The gap is the label delay. A row at time t has a label defined by what happens in the following thirty days, so any training row within thirty days of the validation start has a label that overlaps the validation period — and features computed after t can encode the validation outcomes. Leaving a gap equal to the label horizon between training end and validation start removes both.
Shuffling destroys all of this in one line. Any split that does not preserve order lets the model train on the future of the rows it is validated on, and the metric it produces is an upper bound with no known relationship to production.

What the shuffle let the model see

A random split on customer-weeks builds a validation set whose every row has a near-duplicate in training — the same customer, one week earlier or later — and whose every period has training rows from after it. The model is validated on the past with knowledge of the future and on customers it has memorised. Neither is available in production.

The leak is not a bug in a feature. It is the split itself. No feature audit will find it, because every feature is legitimate; what is illegitimate is the set of rows the model was allowed to train on.

leakagethe training set's temporal extent — rows from after the validation periodThe split as the leak

looks like A conventional shuffled holdout on a table of customer-weeks; no future-dated columns, no target-derived features, clean schema.

why it leaks Training rows from later periods carry the effect of events — a pricing change, a competitor — that the validation rows precede; and adjacent weeks of the same customer sit on both sides, so the model recognises customers instead of predicting churn.

offline

Validation looks excellent and stable across seeds — the leak is systematic, so it reproduces.

production

Every prediction is about the genuine future for customers whose latest week is the last one the model has; performance falls to near the old rule within a month.

fix Forward-chained folds with a gap equal to the thirty-day label horizon; features computed as of each row's cutoff; the most recent period reserved as the untouched test.

when this feature is fine When rows are genuinely independent and exchangeable — no repeated entities, no temporal drift, labels that do not depend on later events — a random split is honest and forward-chaining has nothing to chain. That is rare in business data and should be demonstrated, not assumed.

Forward folds, with the gap

Each fold trains on everything before a cutoff and validates on a window that starts one label-horizon after it. The gap removes the training rows whose labels overlap the validation window, and the as-of feature computation removes the aggregates that would otherwise reach across the cutoff. The result is a rehearsal of a deployment at that date.

Moving the cutoff forward produces a sequence of rehearsals. The sequence is the evaluation: not only how good the model is, but how quickly it stops being that good.

Forward-chaining folds with a label-horizon gap

1import numpy as np
2 
3def forward_folds(ts, cutoffs, horizon, window):
4    """ts: timestamp per row.  For each cutoff c:
5       train  = rows with ts <  c
6       gap    = rows with c <= ts < c + horizon   (labels overlap validation: dropped)
7       valid  = rows with c + horizon <= ts < c + horizon + window
8    """
9    for c in cutoffs:
10        tr = ts < c
11        va = (ts >= c + horizon) & (ts < c + horizon + window)
12        yield tr, va
13 
14# thirty-day horizon, four-week validation windows, cutoffs every quarter
15# every feature for a row at time t must be computed from data with ts <= t
16for tr, va in forward_folds(ts, cutoffs, horizon=np.timedelta64(30, 'D'),
17                            window=np.timedelta64(28, 'D')):
18    ...   # fit on tr, score on va, keep the sequence of scores — do not average it away

The gap is the line most teams delete to "keep more data". Keep it. Its width is the label horizon, and a row inside it has a label that was decided by events the validation window contains.

The serving path has to keep the promise

Forward-chaining makes a promise: every feature was computed from data available at the row's timestamp, with the delays that existed then. The serving path has to keep it. If the feature pipeline in production reads a table that is backfilled, or an aggregate that includes today's late-arriving events, the honest validation described a system that production does not run.

So the split discipline is also a monitor: the most recent forward fold predicts the first month of production, and a shortfall is either drift or a feature that production computes differently from the validation. Both are worth knowing, and the fold is what makes either visible.

Churn model, random split to production

offline evaluation said

Random shuffled holdout: excellent ranking, stable across seeds. The forward-chained version, run afterwards, is markedly weaker on the most recent fold and falling across folds after the pricing change.

production did

First month of production, labels arrived: performance matches the most recent forward fold, not the random split; the retention team saw the gap as the model "breaking".

What explains the gap — most likely first

1The random split let the model recognise customers and learn the pricing change from the future; production offered neither.
2The forward folds showed the metric falling after the pricing change — a drift signal that was available before deployment and not read.
3A support-contact feature was computed from a table that is backfilled weekly, so even the forward folds slightly overstated what the serving path could compute at prediction time.

what it costs to close or detect The honest number is lower and arrives with a trend that has to be explained to stakeholders who saw the higher one first. The gap costs a label-horizon of training data per fold. And the as-of feature computation is a rewrite of the feature pipeline, which is the real price of temporal validation and the reason it gets skipped.

How to build it

Most important first.

Split by time, always. Training ends at a cutoff; validation begins after the cutoff plus the label horizon; the test period comes after that, untouched (Time-Based Split).
Chain forward. Several cutoffs, each producing a train/validation pair, so the metric is a sequence rather than a point and its trend is visible (Forecast Evaluation).
Build features as of the cutoff only. Every aggregate a row carries must be computable from data available at that row's timestamp, which is the point-in-time discipline the serving path will have to reproduce (Point-in-Time Correctness).
Decide whether the training window is expanding or rolling. Expanding uses all history and averages across regime changes; rolling uses recent history and tracks them. The choice is a bet about how fast the world moves, and forward-chaining lets you test it.

What to measure

Which number actually maps to the decision — and which numbers look relevant and are not.

The validation metric on the most recent forward fold — the one that most resembles the deployment — and its trend across folds, which is the drift estimate.
The gap between random-split and forward-chained metrics, once, as a measurement of how much the random split was leaking.
Do not average the forward folds into a single number and lose the trend. The trend is what the folds were for.

What must stay true after deployment

The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.

Assumptions

The cutoff discipline used in validation is the discipline the serving path enforces — every feature at prediction time is computed from data that existed then, with the same delay the validation gap assumed.
The label horizon has not changed. A change in the definition of churn, or in how long a cancellation takes to be recorded, changes the gap the split needs.
The rate of change in the world stays within what the forward folds showed; a regime change larger than any in the two-year history is outside what the validation could estimate.

How to verify — offline, online, and over time

Offline: run the random split and the forward-chained split on the same features. The difference is the leak. Then run forward folds with and without the gap; the difference is the label-horizon leak.
Online: compare the first month of production performance, once its labels arrive, against the most recent forward fold. They should agree; if the fold was better, look for a feature the serving path cannot compute at prediction time.
Over time: add each new month as a fold and keep the sequence; a fold that drops sharply is the earliest available drift signal, arriving with the labels.

What can go wrong

Failure modes in production

The gap is set to zero because "we lose a month of data"; training rows just before the cutoff carry labels that overlap the validation window and the leak returns, smaller and harder to see.
A feature is computed from a table that is periodically backfilled, so the training row for week t contains a value that was corrected in week t+3; the forward fold looks honest and the feature is still from the future (Validating a Backfill Before You Publish).
The most recent fold is the only one reported because it is the best; the earlier folds, which showed the metric falling after the pricing change, are not shown and the drift is discovered in production.

What the recommended approach costs

A temporal split leaves less data in each training fold than a random one, and the most recent fold — the one that matters most — has the least validation data of all.
The gap discards a label-horizon of training data at every cutoff, which for a thirty-day horizon on two years of history is not much and for a one-year horizon is most of it.
Forward-chaining is k trainings on growing windows, which is slower than k equal folds and cannot be parallelised as cleanly.

Misreads

"Random split works for everything if the data is large enough." Size does not remove order. A million shuffled customer-weeks leak exactly as a thousand do; the model recognises customers and learns the future either way.
"We have a time-based split, so we are safe." Only if the features respect it too. A feature computed from a backfilled table, or an aggregate that includes rows after the cutoff, leaks through a perfectly ordered split.
"The forward folds are worse than the random split, so the temporal split is pessimistic." It is accurate. The random split was an upper bound produced by leakage. The forward-fold number is the one production will resemble.

Where this applies

ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

GENERALThat a model predicting the future must be validated on the future, with a gap for the label delay, holds for every temporal prediction task regardless of model family.
TASK-SPECIFICChurn, fraud and demand are temporal by construction; a task with no time structure — classifying images with no drift and no per-entity repetition — has nothing to chain and a random split is honest. The mistake is assuming a task is the second kind because the dataset arrived as a flat table.
CONTESTEDA serious position holds that strict forward-chaining is too conservative for slowly-changing domains: it throws away data at every gap, gives the most important fold the least validation data, and the pessimism it introduces leads teams to under-invest in models that would have worked. The reply is that the pessimism is measured — run both splits and the difference is the leak — and that a slowly-changing domain shows a small gap, which is the evidence that a looser split would be safe, not an assumption to start from.

Where the depth lives

This domain teaches the model and hands the rest off by name.

Data Engineeringevent-time late-arriving-data backfill-validation

Domains that do not exist yet

— Time-series econometrics — rolling-origin evaluation, the choice between expanding and rolling windows, and the statistical comparison of forecasts across origins are treated formally in the forecasting literature.