Feedback Loops
The model decides what users see, what users see decides what they click, and what they click is the next training set. Retraining on that log does not correct the loop — it tightens it.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
How does a recommender shape its own training data, why does naive retraining make the bias worse, and what breaks the loop?
A video platform's recommendations have converged. The content team reports that the same few hundred videos dominate the home page across every audience, new creators cannot get views, and retention has been sliding for two quarters. "We retrain weekly on the latest data," the ML team says. "The metrics keep improving."
Retrain weekly on the latest log. Fresh data, fresh model, metrics improve every week. The system is learning from more and more behaviour.
Each week's log contains what last week's model showed. The new model learns that those items get clicks — because they were shown — and shows them more. Next week's log has even more of them. The offline metric improves each week because the model is increasingly good at predicting a log it increasingly wrote.
- Each week's log contains what last week's model showed. The new model learns that those items get clicks — because they were shown — and shows them more. Next week's log has even more of them. The offline metric improves each week because the model is increasingly good at predicting a log it increasingly wrote.
- Position bias compounds it: whatever was at the top got more clicks, the model learns those items are good, it puts them at the top again. The click was about the position and the model attributed it to the item.
- Popularity amplifies until the head of the catalogue is a fixed set and the tail is unreachable; new items enter cold and stay cold (Cold Start).
- Retention slides because the feed is the same for everyone and stops surprising anyone, and nothing in the offline evaluation can see it: the evaluation is on the log, and the log is the loop.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The model predicts engagement on an item it shows. The system's target is retention. The two have been diverging for two quarters and the model's offline metric has not noticed.
- The training label is a click or a watch on an impression the previous model produced. It is a label about the previous model's choices as much as about the user.
- Weekly impression logs. Every row is an item the previous model chose to show, at a position the previous model chose, and what happened. Items the model did not show do not appear.
- Position is logged but not used; propensity — the probability the model assigned to showing this item — is not logged at all.
- There is no randomised slice. Every impression for two years has been chosen by some version of the model.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- The loop is Model → User Experience → User Behaviour → Training Data → Model. Each arrow is a dependency: the experience is what the model chose, the behaviour is conditioned on the experience, the training data is the behaviour, and the next model is fitted to it. A closed loop with a positive gain amplifies whatever it starts with.
- This is selection bias with the model as the selector (Selection Bias). The training set is a non-random sample of (user, item) pairs, and the sampling mechanism is the previous model's preference. A model trained on it estimates "engagement given the old model showed it", not "engagement if shown".
- Naive retraining reinforces the loop because the new model has no evidence about anything the old model did not show. Where there is no counter-evidence, the prior — which is the old model's choices — wins, and the bias compounds with every iteration.
- Breaking the loop requires data the model did not select: a randomised slice, an exploration slot, a held-out control group — and a way to use it, which is propensity weighting or counterfactual evaluation.
The loop, drawn
Five boxes and five arrows, and every arrow is a dependency. The model chooses the experience; the experience conditions the behaviour; the behaviour is logged as training data; the training data determines the next model. A loop with positive gain amplifies its initial state, and the initial state is whatever the first model preferred.
The model in this picture is not observing the world. It is observing its own reflection, and the reflection is what it learns from.
Why the training set only contains what the last model showed
A row in the impression log exists because the previous model put that item at that position for that user. An item the model did not show has no row. The training set is therefore a sample selected by the previous model, and the selection is correlated with the label — the model showed things it expected to be clicked.
This is selection bias, and the standard correction is to weight each example by the inverse of the probability it was selected. That probability is the propensity, and it has to be logged at serving time; it cannot be reconstructed afterwards, because the serving policy that produced it is gone.
looks like The obvious item-quality signal: items with a high CTR are items people like.
why it leaks Most of an item's impressions were at the positions the previous model gave it. A high-CTR item is mostly an item that was shown high; the feature encodes the previous model's ranking decisions and the layout's position effect, and the model learns to reproduce them.
fix Model position explicitly and estimate item quality with position held out — CTR at a fixed position, or a position-debiased estimate from the randomised slice — and log propensities so the estimate can be corrected.
Breaking the loop, and what it costs
Every mitigation injects data the model did not select. Randomised exploration slots produce impressions with known, model-independent propensities. Counterfactual evaluation uses that slice to estimate what a candidate model would have achieved without shipping it. A control group measures the outcome the model cannot corrupt.
None of them is free and none of them is a modelling change. They are decisions about what the product shows and to whom, and they are argued against by the offline metric every week, because the offline metric is on the loop's side.
Each week's retrained model beat the previous week's on the held-out slice of that week's log, consistently, for two quarters.
Impression concentration on the top few hundred items rose every week; new-creator views fell; retention in a late-added control group ran above the model arm.
- 1The held-out slice is the previous model's output, so each new model was rewarded for agreeing with its predecessor more precisely.
- 2Position bias attributed the layout's effect to the items at the top, which were then placed at the top again.
- 3No randomised data existed, so no candidate model could ever be credited for an item the incumbent had not shown.
A known fraction of impressions were chosen independently of the model, their propensities were logged as served, and they are present in training and evaluation with those propensities.
holds when The randomised slice runs continuously and covers the tail; re-ranking rules after the model are reflected in the logged propensity; the slice is never excluded from training as noise.
breaks when The slice is switched off or shrunk; a downstream rule overrides the model without updating the propensity; a new surface launches without the slice.
respond Restore the slice and the propensity contract before any retraining. A retrain on a log with no independent exposure reinforces the loop, whatever else changed.
How to build it
Most important first.
- Log the propensity: for every impression, the probability the serving policy assigned to showing that item at that position. Without it, no correction is possible after the fact.
- Run a randomised exploration slice — a small share of impressions chosen uniformly or from a broad prior — continuously, and keep it in the training set with its propensity.
- Weight training and evaluation by inverse propensity, so an impression the model was unlikely to show counts for more; evaluate candidate models counterfactually on the randomised slice before any online test.
- Hold out a control group that does not receive the model at all, or receives a frozen one, and compare retention against it over quarters. It is the only measurement the loop cannot corrupt.
- Monitor the loop's signature directly: impression concentration, catalogue coverage, and agreement between consecutive models, weekly (Model Monitoring).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Retention in the control group against the model arm, over a horizon long enough for the feed's monotony to matter. This is the number that says whether the loop is hurting.
- Impression concentration and coverage per week: the share of impressions on the top 1% of items and the fraction of items with any impression. These rise and fall respectively as the loop tightens, and they do so before retention moves.
- Do not measure week-over-week offline improvement on the log. It improves *because of* the loop, and reads as progress.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Some impressions are chosen independently of the model, and their propensities are known, so the training set contains evidence the model did not create.
- The logged propensity is the probability the serving system actually used — re-ranking rules, business overrides and fallbacks after the model did not change it without being logged.
- The control group is protected: nobody borrows it for a launch, and its users are not exposed to the model through another surface.
- Offline: counterfactual estimates of a candidate model's engagement on the randomised slice, with confidence intervals; agreement rate between the candidate and the incumbent on what they would show.
- Online: the control-group retention comparison; an A/B test for any candidate whose counterfactual estimate looks better.
- Over time: the weekly concentration and coverage series with a threshold that pages; the Drift Explorer at
/ml/drifthas a feedback-loop scenario in which feature drift appears with no external cause because the model shifted its own inputs — the pattern to recognise (Drift Is Not Failure).
What can go wrong
- The exploration slice is cut to recover a quarterly engagement number; the propensity logging stays, but every propensity is now from the model and the correction has nothing to correct with.
- Inverse-propensity weights are clipped hard to control variance, which quietly removes the correction for exactly the low-propensity items where it matters.
- The control group is small enough that retention differences are inside the noise, and the loop is declared harmless.
- Counterfactual evaluation is used to choose among candidate models and then the winner is shipped without an online test; the estimator's variance on the tail was never checked.
- Exploration costs engagement on every impression it takes, and it never stops costing. The benefit is data quality, which shows up in the next model and is owned by nobody.
- Propensity logging is a schema change to the busiest event stream and a contract with every re-ranking rule downstream of the model.
- A control group is a share of users who receive a worse product indefinitely so the rest can be measured; the ethics and the product politics are real.
- "Metrics improve every retrain, so the system is learning." It is learning to predict a log it wrote. The offline metric on the loop's own data cannot distinguish learning from self-agreement.
- "Drift means retrain." When the drift is the loop — the model shifted its own input distribution — retraining on the shifted data tightens the loop. The Drift Explorer's feedback scenario is a drift that retraining makes worse.
- "Position bias is a small correction." It is the mechanism by which the loop attributes the layout's effect to the item. Uncorrected, it makes whatever was on top last week correct this week.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe loop exists wherever a model's decisions select the examples it will later train on: recommendation, search ranking, ad selection, fraud review queues, hiring screens, credit approval. The mitigations transfer; the cost of exploration differs by domain.
- DOMAIN-SPECIFICIn media the cost of an exploratory impression is a slightly worse feed; in credit or hiring an exploratory decision is a loan or a job given to someone the model would have rejected, and the data-quality argument has to be made against a real cost to real people.
- CONTESTEDA serious position holds that heavy propensity machinery is over-engineering for most teams, and that a modest randomised slice plus a protected control group catches the loop well enough without inverse-propensity training, whose weights are high-variance and easy to get wrong. The counter is that a randomised slice too small to cover the tail leaves the correction undefined exactly where the loop does its damage, and that the control group measures harm only after it has happened.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Causal inference — inverse propensity weighting and counterfactual (off-policy) evaluation are estimators with their own variance and assumptions, and the statistics of when they can be trusted are a discipline this lesson relies on without teaching.