Offline vs Online Evaluation
Offline evaluation scores a model on a historical dataset produced by the previous policy. Online evaluation measures what happens when the model acts on live traffic. Strong offline numbers are a reason to run the online test, not a substitute for it.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Offline, the new recommendation model beats the incumbent on every metric. What could make it worse in production, and what would you have to run to find out?
A streaming service's recommendation team has a new ranking model that scores better on every offline metric they track. The last two models that looked this good produced no measurable change in watch time when released, and one reduced it. Leadership wants to know why offline results keep failing to predict production, and whether to bother with offline evaluation at all.
Hold out the most recent month of logs, score both models on it, and ship the one with the better ranking metrics. The held-out set is real user behaviour, so a model that ranks it better will rank production better.
The held-out logs contain only titles the old model showed. The new model is rewarded for agreeing with the old model about which titles get watched and cannot be rewarded for the titles it would show instead, because those have no labels. The offline metric measures similarity to the incumbent as much as quality.
- The held-out logs contain only titles the old model showed. The new model is rewarded for agreeing with the old model about which titles get watched and cannot be rewarded for the titles it would show instead, because those have no labels. The offline metric measures similarity to the incumbent as much as quality.
- The new model's features are computed offline from complete logs; in production they are computed from a cache that is minutes stale for the busiest users. The model that was evaluated is not the one serving (Train / Serve Skew).
- The label is a click or a short watch. The business metric is retention. The new model is better at getting a click and worse at getting a satisfied user, and the offline metric cannot tell the difference because it was built from the proxy.
- Once released, the new model changes what users see, which changes what they watch, which changes the next training set. The offline number was computed on a world the model itself will alter (Feedback Loops).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Rank candidate titles for a user so that the ones they will watch appear first. The label is a watch event on a title that was shown — a title never shown cannot have a label.
- The business outcome is watch time and retention over weeks; the offline proxy is whether a shown title was clicked or watched, over a logged session.
- One example is one impression: user, title, position, context, and whether it was watched. Billions of rows, all generated by the incumbent model choosing what to show.
- The held-out set is a later slice of the same logs. Every positive label in it is a title the old model chose to surface; the new model's preferred titles, where they differ, have no labels at all.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Offline evaluation is a computation over a fixed dataset: scores from the candidate model, labels from history, a metric between them. Its validity depends on the dataset being a fair sample of what the model will face and the labels being the outcome the business cares about. Both are assumptions, and both are usually false in some degree.
- The dataset was produced by the previous policy — the old model, or the old rule — which decided which examples exist and which have labels. This is the selection-bias structure of every logged-interaction dataset: the new model is scored on the old model's choices. The further the new model departs from the old, the less the offline metric can say about it.
- Online evaluation replaces the fixed dataset with live traffic: the candidate model acts, the outcome is observed, and the comparison is against a control arm acting on the same traffic. It measures the business metric directly, on the population the model creates, with skew and drift included. It is expensive, slow, and the only evaluation that is about production.
The dataset was made by the last model
Every impression in the logs exists because the incumbent decided to show that title. Every label is a reaction to that choice. A new model that would show different titles is being scored on whether it agrees with the incumbent about the titles the incumbent showed — and where it disagrees, there is nothing to score.
This is why the offline number flatters models that resemble the incumbent and cannot see the value of models that do not. The two previous releases that looked good offline and did nothing online may simply have been small perturbations of the old model, rewarded for agreeing with it.
looks like A clean, recent, held-out month of real user behaviour with watch labels, split by time, no obvious leakage.
why it leaks The labelled population was selected by the old model. The candidate is scored only on titles the old model surfaced, so the metric rewards agreement with the old policy and is silent about the candidate's own choices.
fix Log a randomised fraction of impressions so the held-out set has labels independent of the incumbent, evaluate on that slice, and treat the full-log metric as a similarity score.
What the online test measures that the offline one cannot
The online experiment puts the candidate in front of live users with its production features, its production latency and its own choices, and observes the business metric on the population it creates. Every mechanism the offline evaluation could not see — skew, drift since the held-out month, the feedback loop, the gap between click and retention — is inside the measurement.
That completeness is also why it is slow and expensive: the business label takes as long as it takes, the control arm keeps running the old model, and the experiment consumes traffic that cannot be used for the next candidate.
Better than the incumbent on every ranking metric on the full held-out logs, by a comfortable margin; slightly better on the small randomised-impression slice, within noise.
Click-through up in the first week; watch time flat; retention at four weeks slightly down in the treatment arm, with an interval that includes zero.
- 1The candidate departs from the incumbent more than previous releases did, so the full-log metric — which rewards agreement — overstated it, and the randomised-slice metric, which did not, was the honest and unimpressive number.
- 2The proxy label is a click; the candidate is better at producing clicks and no better at producing satisfying watches, and retention follows the second.
- 3Serving features for heavy users come from a cache that lags by minutes, which the offline evaluation, computed from complete logs, never saw.
- 4The four-week retention label had barely arrived when the first read was taken, so the early read was dominated by the click effect.
A ladder, not a choice
Offline and online evaluation are not competitors. They are rungs: the offline metric on the full logs filters out candidates that cannot even agree with the incumbent; the randomised-slice metric filters candidates that agree without improving; shadow mode catches skew and latency; the controlled experiment measures the business outcome. Each rung is cheaper than the next and less about production.
The failure is skipping rungs in either direction — shipping on the first because it was good, or running the last on every candidate because the first "cannot be trusted". The ladder is what makes the expensive rung affordable.
- 1Offline, full logs
Rank candidates by the standard metric on the held-out month.
fails by Rewarding agreement with the incumbent; a candidate that would change what is shown scores badly for the wrong reason.
- 2Offline, randomised slice
Re-score the survivors on impressions the incumbent did not choose.
fails by Too few rows to separate candidates; the team falls back to the full-log number.
- 3Shadow
Score live traffic with production features and latency, without acting.
fails by Sharing the skew with the future serving path, so both are wrong together.
- 4Controlled experiment
Act on a traffic split; compare the pre-registered business metric against control after the label window.
fails by Being stopped early on the proxy, or having the intervention changed mid-run.
- 5Release and re-evaluate
Ship, and keep the randomised slice flowing to recalibrate the offline-to-online gap.
fails by The slice being switched off to reclaim the traffic, and the next offline number being trusted blind.
How to build it
Most important first.
- Use offline evaluation to decide what is worth testing online, not what to ship. A candidate that does not beat the incumbent offline is not worth an experiment; one that does has earned one (Beating the Baseline).
- Reduce the policy-dependence of the offline set where it is cheap: log a small fraction of randomised impressions so the held-out set contains labels the old model would not have produced, and weight the evaluation accordingly.
- Before the experiment, run the candidate in shadow — scoring live traffic without acting on it — to catch skew and latency problems that the offline evaluation could not see (Shadow Deployment).
- Run the online test with a control arm, a pre-registered business metric, a duration set by the label delay and the effect size, and a guardrail metric that can stop it (A/B Testing Models).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Watch time and retention in the treatment arm against control, over a window long enough for the retention label to exist. This is the release criterion.
- Offline ranking metrics on the randomised-impression slice, as the pre-experiment filter — the closest offline number to production, and still not production.
- Do not use the offline metric on the full logs as a release criterion. It measures agreement with the incumbent on the incumbent's choices.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The held-out dataset is a fair sample of the impressions the candidate model will actually produce, or has been corrected for the incumbent's selection — otherwise the offline metric measures agreement with the old policy.
- The proxy label the offline metric is computed on moves in the same direction as the business metric; a click that predicts retention today still predicts it after the model changes what is clickable.
- The features and latency in the offline evaluation match those in serving, so the model being evaluated is the model being deployed.
- Offline: compare the candidate on the full logs and on the randomised-impression slice. A large gap between the two means the candidate departs from the incumbent and the full-log number is unreliable in proportion.
- Online: shadow first, for skew and latency; then a controlled experiment on the business metric with a pre-registered duration and stopping rule.
- Over time: keep the randomised slice flowing and re-evaluate the shipped model on it monthly; the offline-to-online gap for past releases is the calibration for how much to trust the next offline number.
What can go wrong
- The randomised impressions are too few to compute a stable offline metric, so the team quietly falls back to the full logs and the filter stops filtering.
- The online experiment is stopped early because the click metric moved; the retention metric, which had not arrived yet, moves the other way, and the model ships on the proxy it was supposed to be checked against.
- Shadow mode passes because the shadow path uses the same stale cache as the future serving path, so the skew is present in both and invisible in neither.
- Randomised impressions cost the experience of the users who receive them, and the fraction that makes the offline metric usable is a fraction of traffic showing worse recommendations on purpose.
- Online experiments take as long as the business label takes to arrive, which for retention is weeks; the release cadence is bounded by the label delay, not by training time.
- Shadow deployments double the serving cost for the duration and still cannot see what happens when the model's output is acted on.
- "If offline AUC improved, ship it." Offline improvement on policy-dependent logs is the entry ticket to an experiment. Shipping on it means shipping on agreement with the old model about the old model's choices.
- "Offline evaluation is useless; just test everything online." Online tests are slow and expensive and each one burns traffic. Offline evaluation is what makes the online budget affordable by filtering candidates; it is a prior, not a verdict.
- "The last model failed online, so the offline pipeline is broken." The offline pipeline may be fine and the gap may be structural: skew, a proxy label, or selection by the incumbent. Diagnose which mechanism before rebuilding the metric.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat the offline dataset was produced by the previous policy and that the online outcome is the only measurement of production holds for every deployed model that influences the data it is later evaluated on.
- TASK-SPECIFICRecommendation and ads make the selection problem extreme because the model chooses what gets labelled; a medical risk model scored on outcomes that occur regardless of the prediction has a much smaller offline-to-online gap from this cause, and a larger one from skew and drift.
- CONTESTEDA serious position holds that counterfactual evaluation — inverse propensity weighting, doubly robust estimators over logged bandit feedback — can close most of the offline-to-online gap without spending traffic, and that teams which run an online test for every candidate are wasting their experiment budget on questions a good off-policy estimator answers. The reply is that off-policy estimators need logged propensities and enough overlap between policies to have acceptable variance, and most teams have neither.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Causal inference — off-policy evaluation, inverse propensity weighting and the conditions under which logged bandit feedback can estimate a new policy's value are the formal treatment of the selection problem this lesson describes.