Candidate Generation vs Ranking
Millions of items cannot be scored by a rich model inside a page-load budget. Retrieval narrows to hundreds with a cheap model; ranking orders them with an expensive one; each stage has its own metric and its own way to fail.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Why is a recommender split into retrieval and ranking, how is the latency budget divided, and why do the two stages need different offline metrics?
An e-commerce team built a strong ranking model with two hundred features. It scores a (user, item) pair in a millisecond. The catalogue has three million items and the page has to render in under two hundred milliseconds. "The model is great," they say. "We just cannot run it."
Score everything. If it is too slow, cache the scores per user nightly, or add machines. The ranking model is the thing that works, so make it possible to run.
Nightly per-user caching of three million scores is a matrix the size of the catalogue times the user base, and it is stale for the whole day: a user who bought a tent this morning is shown tents all afternoon.
- Nightly per-user caching of three million scores is a matrix the size of the catalogue times the user base, and it is stale for the whole day: a user who bought a tent this morning is shown tents all afternoon.
- Adding machines does not turn a millisecond per pair into three million pairs in a hundred milliseconds. The arithmetic does not close by an order of magnitude, whatever the budget.
- A cheap first stage is added as a filter — "top ten thousand by popularity" — and the ranker's offline metric barely moves, but the items it never sees are precisely the ones a popularity filter drops, and the ranker's quality on the long tail was never exercised.
- The ranker was trained on impressions from the old system, which were selected by the old retrieval. When retrieval changes, the ranker sees candidates from a distribution it was not trained on and its scores on them are unreliable.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The system's target is the top few items for a user out of millions, judged by what the user does with them. The retrieval stage's target is *not to lose* those items; the ranking stage's target is to *order* them.
- The two targets are different enough that a metric for one says little about the other.
- Item embeddings from a collaborative or two-tower model, precomputed nightly and stored in an approximate nearest-neighbour index (Collaborative Filtering).
- A ranking training set: logged impressions with rich features — user history aggregates, item attributes, context, position — and the outcome. Only items that reached the ranking stage appear in it.
- A latency budget for the whole page, of which the recommender gets a share, of which retrieval and ranking each get a share (Latency Breakdown).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Retrieval reduces millions to hundreds using a representation that can be indexed: an item embedding and a user embedding whose dot product is the score, so the top candidates are the nearest neighbours of the user vector in the index. Approximate nearest-neighbour search makes this sub-linear in the catalogue size, at the cost of occasionally missing a true neighbour (Vector Search: Embeddings, Similarity and ANN).
- Ranking takes the hundreds and scores each with a model that can afford cross features — interactions between user and item attributes that a dot product cannot express — because the candidate count is small enough to pay for it.
- The latency budget is split accordingly: retrieval must be fast because it touches the index; ranking must be fast per candidate because it runs hundreds of times; feature fetching for the ranker is often the largest single term and is easy to forget (Feature Freshness).
- Offline metrics follow the jobs. Retrieval is measured by recall@k — is the item the user engaged with in the candidate set at all — because a missed item cannot be recovered. Ranking is measured by an ordering metric such as NDCG over the candidate set, because its job is the order and it cannot add items.
The funnel and its latency budget
The ranker's cost is per candidate. Retrieval's cost is roughly per query, thanks to the index. So the number of candidates is the knob that trades retrieval's recall against ranking's latency, and it is set by the page budget rather than by either model's preference.
Feature fetch is the term teams forget. A ranker with two hundred features needs them for every candidate; if they come from a feature store over the network, that fetch — not the model — is usually the largest share of the budget (Feature Stores).
- 1User embedding
Compute or fetch the user vector for the query.
fails by A cold user has no vector and retrieval falls back to popularity (Cold Start).
- 2ANN retrieval
Nearest neighbours of the user vector in the item index, plus other sources, unioned and deduplicated.
fails by The index is stale or approximate in a way that drops rare items; recall on the tail collapses.
- 3Feature fetch
Gather ranking features for every candidate from the feature store.
fails by Latency scales with candidate count; a slow store makes the ranker score defaults.
- 4Ranking
Score each candidate with the rich model; sort.
fails by Candidates from a new retrieval source are outside the training distribution.
- 5Re-rank
Apply diversity, business rules, exploration slots.
fails by Rules override the model so often that the model's effect is unmeasurable.
Two jobs, two metrics
Retrieval cannot be measured by ordering, because its job is membership: was the item the user went on to engage with in the candidate set at all. Recall@k is the natural metric, and it should be computed against engagements from surfaces that retrieval did not produce, or it measures retrieval against itself.
Ranking cannot be measured by recall, because it cannot add items. Its job is the order of what it was given, so an ordering metric — NDCG is the usual one, weighting the top positions most — over the logged candidate set is the natural measure (Ranking).
Report the ranker's NDCG on held-out impressions and treat it as the recommender's quality.
Retrieval recall@k by popularity band against out-of-surface engagements; ranker NDCG on logged candidate sets; the online business metric for the whole.
The ranker's NDCG is conditioned on what retrieval returned and is silent about what it did not. Retrieval recall says whether the right items were reachable. Neither predicts the online outcome when the two stages change together, which is why the third number is not optional.
The coupling that no per-stage metric sees
The ranker was trained on impressions, and impressions are retrieval's output. Change retrieval and the ranker is handed candidates from a distribution it never saw; its scores on them are extrapolation, delivered with the same confidence as everything else.
The Inference Decision Tool at /ml/inference walks the freshness, precomputability and latency questions that decide whether each stage runs batch, online or as a hybrid — retrieval is typically a precomputed index with an online lookup, and ranking is online because its features are (Choosing the Inference Mode).
The candidate distribution the ranker scores in production is the one its training impressions were drawn from.
holds when Retrieval is unchanged since the ranker's training set was logged, or the ranker was retrained on candidates from the new retrieval before both shipped.
breaks when A retrieval source is added or the index is rebuilt with a new embedding model; a candidate count increase reaches deeper into the tail than training impressions ever did.
respond Retrain the ranker on the new retrieval's candidates before shipping the retrieval change, or ship the retrieval change to a small arm and let the ranker's new impressions accumulate first.
How to build it
Most important first.
- Set the budget first: how many candidates the ranker can afford within its share of latency at the page's peak concurrency, and design retrieval to return that many (Throughput vs Latency).
- Use several retrieval sources — embedding neighbours, recent-history co-occurrence, popularity within the user's segment — and union them, so one source's blind spot is covered by another.
- Evaluate retrieval on recall against engagements from other surfaces (search, direct navigation), so the check is not confined to what retrieval already returns.
- Train the ranker on candidates produced by the *current* retrieval, and retrain it when retrieval changes; the ranker's training distribution is retrieval's output.
- Instrument the boundary: log the candidate set with its source, so an item that was never shown can be attributed to "not retrieved" or "retrieved and ranked low".
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Retrieval: recall@k against engaged items, split by source and by item popularity. A recall number that is high on the head and near zero on the tail says the retrieval is a popularity filter.
- Ranking: an ordering metric over logged candidate sets, with position weighting; and the p99 of ranking latency including feature fetch, which is the number that decides the candidate count.
- System: the online business metric. Neither stage's offline number predicts it; a retrieval improvement that adds candidates the ranker was not trained on can lower it.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The ranker is scored on candidates drawn from the same distribution its training impressions came from — the retrieval that produced the training set is the retrieval in production.
- The item vectors in the ANN index and the item features the ranker reads describe the same version of each item.
- The candidate count the ranker receives stays inside what its latency budget can absorb at peak concurrency, including feature fetch.
- Offline: recall@k for retrieval by popularity band; an ordering metric for the ranker on logged candidate sets; a replay of the ranker on candidates from the new retrieval to check score distributions before shipping either.
- Online: an A/B test of the full system; per-stage latency percentiles at peak; the fraction of requests that fell back at each stage.
- Over time: candidate-set overlap between retrieval versions; drift in the ranker's score distribution when retrieval changes; the index rebuild lag against the feature pipeline.
What can go wrong
- The ANN index is rebuilt nightly from vectors that the ranker's feature pipeline updated this morning; retrieval and ranking disagree about what an item is (Embedding Drift).
- Feature fetch for the ranker times out under load and the ranker scores candidates with defaults; the ordering is garbage and the fallback is invisible because the page rendered (Serving Fallbacks).
- Retrieval is tuned until recall@k peaks, which is achieved by returning more candidates, which blows the ranking latency budget on peak traffic (Tail Latency: Why p50 Being Fine Does Not Help).
- A new retrieval source is added; its candidates are novel to the ranker; the ranker gives them confident scores from extrapolation, and they appear at the top for a week.
- Two models and an index against one model that cannot run — but also two evaluations, two training sets, a boundary where items are lost, and a coupling where a retrieval change silently invalidates the ranker.
- Approximate nearest neighbour is fast because it is approximate; the missed neighbours are usually rare items, which is a bias in the same direction as everything else in the system.
- More candidates improve recall and cost ranking latency linearly; the trade is set by the page budget and there is no free point.
- "Retrieval is just a performance optimisation." It decides what the ranker can ever show. An item retrieval never returns has a score of nothing, however good the ranker is.
- "The ranker's NDCG is high, so the recommendations are good." NDCG is computed over the candidates retrieval supplied. It says nothing about the items that were never in the set.
- "We improved retrieval recall, so the system improved." The new candidates are ones the ranker has never seen in training, and its scores on them are extrapolation. Recall went up; the online metric may not.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- SCALE-SPECIFICThe split is forced by catalogue size against latency budget: with tens of thousands of items and a lenient budget, one model can score everything and a separate retrieval stage adds a boundary for no gain.
- TASK-SPECIFICThe same funnel appears in search ranking with a query-dependent retrieval and in ad selection with an auction after ranking; the metrics change — search recall is against relevant documents, ads add a bid — but the retrieval/ranking coupling is identical.
- SIMPLIFIEDReal systems often have three or more stages — retrieval, pre-ranking with a light model, full ranking, then a re-ranking for diversity and business rules — and the numbers in this lesson (millions, hundreds, top N) describe the shape rather than any particular system.
Where the depth lives
This domain teaches the model and hands the rest off by name.