Collaborative Filtering
Learn from who interacted with what, with no item attributes at all — and inherit every bias in who was shown what, because the missing entries in the matrix are not negatives.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
How does a model recommend items using only interaction history, and what does it silently assume about the interactions it never saw?
An online marketplace has millions of product listings with terrible metadata — sellers write "nice item, fast shipping" — but a rich purchase and browse history. The growth team asks: "can we recommend from behaviour alone, since the descriptions are useless?"
Factorise the matrix. Give each user and each item a vector so that the dot product predicts the entry; treat empty cells as zeros; train by minimising squared error over the whole matrix; recommend the items with the highest predicted score the user has not touched.
Treating empty cells as zeros teaches the model that everything unseen is disliked. Because most cells are empty and most empty cells are for obscure items, the model learns that obscure items are bad — which is popularity bias written into the loss.
- Treating empty cells as zeros teaches the model that everything unseen is disliked. Because most cells are empty and most empty cells are for obscure items, the model learns that obscure items are bad — which is popularity bias written into the loss.
- The factorisation reproduces the concentration in the log: popular items have dense columns and well-estimated vectors, tail items have noise for vectors, and the top of every user's ranking is the same few hundred products.
- Offline evaluation on held-out interactions looks fine, because held-out interactions are also concentrated on popular items and the model predicts popular items. Catalogue coverage in production is a fraction of a percent.
- A new listing has an empty column and therefore a vector that is either the initialisation or an average — it is never recommended, so it never gets interactions, so it never gets a real vector (Cold Start).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict, for a user and an item, the strength of a future interaction: a purchase, an add-to-cart, a click. The label is an observed interaction; its absence is not a label.
- The ranked list per user is what ships; the per-pair score is an intermediate the ranker consumes.
- A user–item interaction matrix: rows are users, columns are items, an entry is a count or a binary flag. It is enormous and almost entirely empty — a typical user has touched a few dozen items out of millions.
- The entries are implicit feedback: nobody rated anything. A click is weak evidence of interest; a purchase is stronger; the empty cells mean "not shown, or shown and ignored, or bought elsewhere" and the data cannot say which.
- The non-empty entries are concentrated on items the site already promoted. The matrix records exposure as much as preference.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Matrix factorisation writes the interaction matrix as the product of a user matrix and an item matrix of low rank. Each user and each item becomes a dense vector — an embedding — and the predicted affinity is their dot product (Embeddings, Embedding Training).
- The vectors are learned from co-occurrence: users who touched the same items get similar vectors, items touched by the same users get similar vectors. No item attribute is used, which is why it works on garbage metadata and fails on a new item.
- Two-tower models are the neural generalisation: one network maps user features to a vector, another maps item features to a vector, and the score is again a dot product. The split into two towers is what makes retrieval by nearest neighbour possible, because item vectors can be precomputed and indexed.
- Implicit-feedback variants weight observed interactions by confidence and treat unobserved cells as weakly negative rather than as zero targets; the weighting is where the "missing is not negative" correction lives, and it is a modelling choice, not a fact the data supplies.
The matrix and what its empty cells mean
A user–item matrix is the entire input. Rows are users, columns are items, and an entry records an interaction. Almost every cell is empty, and the modelling decision that matters most is what an empty cell is allowed to mean.
If empty means "disliked", the model learns from millions of confident negatives that are mostly items the user never saw. If empty means "unknown", the model has very few examples and must be told how much to trust a sampled negative. Neither choice is free, and the data cannot arbitrate between them.
| Item A (promoted) | Item B (promoted) | Item C (niche) | Item D (new) | |
|---|---|---|---|---|
| User 1 | bought | clicked | — | — |
| User 2 | clicked | bought | bought | — |
| User 3 | clicked | — | — | — |
| User 4 | — | clicked | — | — |
1# Naive: every empty cell is a zero target.2# The loss is dominated by unseen items, mostly obscure ones.3def naive_loss(U, V, R):4 return ((U @ V.T - R) ** 2).sum()5 6# Pairwise: for each observed (u, i), sample an unobserved j and ask7# only that i score above j. Empty cells are "less evidence", not "no".8def pairwise_loss(U, V, observed, sample_negative):9 total = 0.010 for u, i in observed:11 j = sample_negative(u) # the sampler is a modelling choice:12 s_i = U[u] @ V[i] # uniform over items => negatives are13 s_j = U[u] @ V[j] # mostly obscure => "obscure is bad"14 total += -log_sigmoid(s_i - s_j)15 return totalThe pairwise loss fixes the zero-target problem and introduces a new one in the sampler. Drawing negatives uniformly from the catalogue teaches popularity bias through a different door; drawing them in proportion to exposure is the usual correction, which requires knowing exposure.
Vectors learned from co-occurrence, and what they cannot know
Factorisation gives each user and item a dense vector such that the dot product predicts the interaction. Items are close when the same users touched them; users are close when they touched the same items. This is an embedding learned from the log, and it carries the log's geometry — including its concentration on whatever was promoted.
The Embedding Explorer at /ml/embeddings makes the geometry concrete and adds a caution that applies here directly: the neighbours you see in a two-dimensional picture of item vectors are not the neighbours the model uses. Decisions about "similar items" should come from the full-dimensional scores, not from the plot (Embedding Projection Caveats).
looks like A natural signal: an item with many interactions is one users like, so weight its observed cells more.
why it leaks Interaction count is mostly exposure count. Items the previous recommender promoted have more interactions regardless of preference, so the weight encodes the promotion history and the model learns to recommend what was recommended.
fix Normalise confidence by exposure — interactions per impression — or by an estimated propensity; evaluate on the randomised slice where exposure was not chosen by the model.
What must stay true for the vectors to keep meaning something
A learned item vector is a summary of who interacted with the item during training. Deploying it assumes those interactions reflected preference, and that the item's audience has not moved. Both assumptions are quietly broken by the recommender itself, which changes who sees the item.
The monitoring that matters is about concentration and coverage rather than about a metric: the loop shows up as a narrowing of what is recommended long before any held-out number moves.
The interactions the embeddings were learned from were driven mostly by what users wanted rather than by what the previous system chose to show, or exposure has been corrected for.
holds when A randomised exposure slice exists and the loss is weighted by propensity; or the surface shows items with roughly uniform exposure, such as search results for a specific query.
breaks when The training log is the recommender's own output with no correction; a marketing campaign floods a set of items with impressions; a bot account touches thousands of items.
respond Correct the exposure weighting and restore or enlarge the randomised slice before retraining; retraining on the uncorrected log reproduces the concentration with fresher weights.
| Option | Quality | Latency | Cost | Interpretability | Data needed | Operational | Note |
|---|---|---|---|---|---|---|---|
| Item-to-item co-occurrence counts | "Users who bought X also bought Y". No training, trivially served, and entirely captive to popularity. | ||||||
| Matrix factorisation (implicit feedback) | Dense vectors, nearest-neighbour retrieval, and a loss whose treatment of missing cells decides everything. | ||||||
| Two-tower with side features | Item and user features enter the towers, which softens cold start; an index to rebuild and two networks to keep in step. |
caveat The quality column is meaningless without saying on which slice: the co-occurrence baseline is often best on the head and useless on the tail, and the factorisation's advantage appears only when exposure has been corrected for. None of the rows says what happens to a new item, which is the question that decides between them in practice.
How to build it
Most important first.
- Model the missing entries explicitly: either sample negatives from unobserved cells with a weight that reflects exposure, or use a loss that compares an observed item against a sampled one rather than regressing every cell to zero.
- Weight training by inverse exposure or inverse popularity so the loss does not simply reward predicting the head of the distribution; check the effect on coverage, not only on the held-out metric.
- Log impressions so that "shown and not clicked" is available as a genuine negative, distinct from "never shown" (Recommendation Systems).
- Evaluate with coverage and a popularity-stratified metric next to the headline number: how well does the model rank within the long tail, not just overall.
- Pair with a content-based signal for items that have no interactions, and let the embedding take over as interactions arrive (Content-Based Recommendation).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- A ranking metric on held-out interactions *stratified by item popularity*: the head-only number hides everything interesting.
- Catalogue coverage — the fraction of items that receive any recommendation in a week — and the Gini of impressions across items. These are the numbers that show the loop tightening.
- Do not measure squared error over the matrix. It is dominated by the empty cells and rewards predicting zero everywhere.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Co-occurrence in the log reflects preference rather than exposure — or the exposure has been corrected for. When the log is dominated by what the site promoted, the embeddings encode the promotion schedule.
- The set of items with enough interactions to have meaningful vectors covers the items the business wants recommended; the tail the model cannot represent is acceptable.
- The user population's taste structure is stable enough that a user vector learned from last month's interactions describes this month's.
- Offline: the held-out ranking metric reported separately for head, torso and tail items, plus coverage. A model that improves the headline number and worsens tail ranking has learned more popularity, not more preference.
- Online: an A/B test on the business outcome with coverage and impression concentration as guardrail metrics.
- Over time: the fraction of the catalogue that has ever been recommended, per month; a shrinking fraction with a stable headline metric is the signature of the loop.
What can go wrong
- Negative sampling draws uniformly from the catalogue, so the negatives are mostly obscure items, and the model learns "obscure means negative" through the sampler instead of through the zeros.
- A popularity-debiasing weight is tuned until the held-out metric peaks, which is the point where the model has just enough popularity bias to match the biased test set.
- The item embeddings drift as interactions arrive, and a retrieval index built last night disagrees with the ranker's vectors from this morning (Embedding Drift).
- A bot or a bulk buyer touches thousands of items and its row drags every item it touched toward one vector; the "similar items" for a niche product become the bot's shopping list.
- Ignoring item attributes is the strength and the weakness: it works on any catalogue with interaction history and it cannot say anything about an item without one.
- Exposure weighting and negative sampling introduce hyperparameters that trade the headline metric against coverage; the best setting for the business is not the one that maximises the held-out number.
- Embedding-based retrieval needs an index that is rebuilt as vectors change, which is an operational component with its own freshness problem.
- "The matrix is sparse, so we need a model that handles zeros well." The matrix is not sparse with zeros; it is sparse with unknowns. The choice of how to treat an unknown is the whole modelling decision.
- "Popular items score highest because they are what people want." They score highest because they have the most data and appear most in the test set. Whether users would prefer something else has not been tested.
- "Clicks are ratings with a smaller scale." A rating is an opinion about an item the user chose to evaluate; a click is a response to an item the system chose to show. The second is conditioned on exposure and the first is not.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- DATA-SPECIFICCollaborative filtering needs dense-enough interaction history; on a catalogue where most items have a handful of interactions, or a product where most users are one-time visitors, the embeddings are noise and a content or popularity approach outperforms it.
- MODEL-SPECIFICThe missing-is-not-negative problem is described here for matrix factorisation and two-tower models; sequence models that predict the next item from a session history sidestep the matrix but inherit the same exposure bias through the sessions they train on.
- CONTESTEDA serious position holds that explicit popularity debiasing is mostly wasted: popular items are popular because many people like them, and a model that down-weights them trades measurable engagement for a coverage number nobody experiences. The counter-argument is that the head-heavy log cannot tell you whether users would prefer tail items, and that the only way to find out is to show some — which is an exploration decision, not a loss-weighting one.
Where the depth lives
This domain teaches the model and hands the rest off by name.