Classification
The model outputs a probability; the product needs a decision. The threshold between them is where the business cost lives, and it is the part that gets defaulted to 0.5.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The model outputs a category or a probability. How does that become an action, and which metric describes the action rather than the probability?
A subscription product manager says: "We want to know which users are about to cancel so the retention team can call them. The team can make about two hundred calls a week."
Train a classifier, evaluate accuracy on a held-out set, threshold the probability at 0.5, and send everyone above it to the retention team. Accuracy is high and the model is "done".
At 0.5 almost nobody crosses the threshold on a rare-positive problem, so the retention team receives a dozen names a week and has capacity for two hundred. The model is correct and useless.
- At 0.5 almost nobody crosses the threshold on a rare-positive problem, so the retention team receives a dozen names a week and has capacity for two hundred. The model is correct and useless.
- Accuracy is high because predicting "will not cancel" for everyone is already right for most user-weeks. The number is a fact about the class balance, not the model (Accuracy Under Imbalance).
- When the threshold is lowered to fill the queue, the calls go to users who were going to cancel regardless, or who were never going to; nobody measured whether a call changes the outcome, only whether the prediction was right (Causality vs Prediction).
- The users the team called and retained now appear in next month's training data as non-cancellers with strong cancellation features, and the model learns that those features predict staying.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a user will cancel within the next 30 days. The label is the cancellation event from billing; it is observed for every user after 30 days, which makes it a rare, clean, delayed label.
- The decision is who gets a call, with a fixed budget of two hundred per week. That makes the output a ranked list cut at two hundred, not a probability compared against 0.5 (Decision Before Model).
- One example is one user-week: usage features over the previous weeks, tenure, plan, support contacts, and whether a cancellation occurred in the following 30 days.
- Cancellation is rare — a few percent of user-weeks — so a random split has few positives in validation and accuracy is dominated by the negatives (Class Imbalance).
- A user appears in many weeks, so a random split places the same user on both sides (Entity Leakage, Group Split).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A probabilistic classifier outputs a score that, if calibrated, estimates the probability of the positive class given the features (Sigmoid & Probability, Calibration). The score is not a decision. A decision requires a threshold, and every threshold produces a different confusion matrix with different counts of each kind of mistake (The Confusion Matrix).
- The threshold is chosen from the costs. A false positive is a wasted call; a false negative is a lost subscriber. When the budget is fixed, the threshold is whatever score ranks two hundredth, and the quantity to optimise is precision within the top two hundred (Threshold Selection).
- Metrics that summarise across all thresholds — ROC AUC, PR AUC — describe the ranking, not the decision at the operating point. They are useful for comparing models and useless for telling the retention team how many of their calls will be to real churners (ROC AUC, PR AUC).
The threshold is the decision
The classifier outputs a score. Above some value the user gets a call; below it they do not. Every choice of that value produces a different set of calls and a different set of mistakes, and the confusion matrix is the record of those mistakes at one threshold.
The retention team's budget is two hundred calls, so the threshold is whichever score is ranked two hundredth this week. That number will move with the score distribution; what matters is what is inside the two hundred.
Accuracy on this matrix is dominated by the 9652 true negatives and would barely move if the model were replaced by a random ranking. Precision inside the two hundred, and how many of the 210 cancellations the cut captured, are the numbers the team lives with.
Predicting cancellation is not preventing it
The model answers "who will cancel". The business is spending the budget to make that not happen. Those are different questions: some users will cancel whatever the team says, and some were staying anyway. The call helps only in the middle.
The offline metric cannot see this at all. It grades the prediction. Only a randomised holdout — high-scored users the team deliberately does not call — can measure whether the call does anything.
Strong ranking on a user-grouped, time-based holdout; precision at the top-200 cut well above the base rate.
Retention among called users is barely different from the randomised holdout; the retention team reports the same names every week; next month's retrain scores the called-and-retained users as low risk.
- 1The top of the ranking is dominated by users whose cancellation is already decided — a card that will fail, a company that closed — and a call does not change it.
- 2There is no suppression window, so the same users are called weekly and the budget is spent on a fixed set.
- 3Retained users entered the training data as non-cancellers with cancellation features, teaching the model that those features predict staying.
Applying a threshold, and logging it
The serving code turns a score into an action. It should carry the threshold — or the cut — as data with a version, log both the score and the decision, and record the intervention so the next training set can tell the two outcomes apart.
The assumption is that the top of the ranking is still where cancellations are and that a call still changes them, and both are checked from the logs.
1type Scored = { userId: string; score: number; modelVersion: string }2 3export function selectForCall(scored: Scored[], budget: number, suppressed: Set<string>) {4 const ranked = scored5 .filter((s) => !suppressed.has(s.userId)) // called in the last N weeks -> skip6 .sort((a, b) => b.score - a.score)7 const cutScore = ranked[budget - 1]?.score ?? Infinity8 const chosen = ranked.slice(0, budget)9 // holdout: a random slice of the chosen is NOT called, so the effect of a call can be measured10 const holdout = new Set(chosen.filter(() => Math.random() < 0.1).map((s) => s.userId))11 for (const s of chosen) {12 log({ ...s, cutScore, decision: holdout.has(s.userId) ? 'holdout' : 'call', week: currentWeek() })13 }14 return chosen.filter((s) => !holdout.has(s.userId))15}The cut score is logged so calibration compression is visible even though the cut still fills. The holdout is logged so the training pipeline can build an intervention flag; without it the next model learns that being called predicts staying.
How to build it
Most important first.
- Fix the decision first: two hundred calls a week means the output is the top two hundred by score, and the metric is precision at that cut, plus how many cancellations the two hundred captures (Precision, Recall & F1).
- Split by user and by time so the validation set contains users and weeks the model never saw (Time-Based Split, Group Split).
- Measure whether a call changes the outcome with a randomised holdout among the top-scored users, because the model predicts cancellation and the business wants to prevent it (A/B Testing Models).
- Record the intervention in the data so the next training set can distinguish "did not cancel" from "did not cancel because we called" (Feedback Loops).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Precision at the top two hundred per week, on live outcomes after 30 days, and the retention lift among called users against the randomised holdout. Those two numbers map to the decision.
- Recall at the cut says how many cancellations the budget can reach; it sets expectations for the product manager rather than driving the model.
- Accuracy and ROC AUC on a random split are not the product metric: accuracy is dominated by negatives and AUC describes the whole ranking, most of which the team will never call.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The users ranked in the top two hundred are still the ones most likely to cancel, as checked by precision at the cut on live outcomes each month.
- A call still changes the outcome for high-scored users, as measured against a maintained randomised holdout, so the decision the model feeds is worth making.
- The training labels distinguish outcomes where the retention team intervened from those where it did not, so the model does not learn that intervention features predict retention.
- Offline: precision and recall at the two-hundred cut on a time-based, user-grouped holdout; a calibration curve on the same holdout if the score is also shown to humans.
- Online: precision at the cut on realised cancellations, weekly, with a 30-day lag; retention lift against the randomised holdout.
- Over time: a monthly check that the intervention flag is present in the training data and that the score distribution at the cut has not compressed.
What can go wrong
- The top two hundred are the same users every week — high-risk, low-usage — and the team calls them repeatedly; the queue needs a suppression window the model does not know about.
- Calibration drifts after a pricing change and the scores compress; the top two hundred is still a top two hundred, so the cut does not notice, but the precision within it drops.
- The randomised holdout is removed to "stop wasting high-risk users", and from then on there is no way to tell whether the calls do anything.
- A randomised holdout among the highest-risk users means deliberately not calling people the model says will cancel; it is the only way to know the calls work.
- Optimising for precision at a fixed cut ignores everything below it, so a model that ranks the middle better gets no credit.
- Recording interventions in the training data requires the retention team's tooling to write back, which is an integration nobody planned.
- "Accuracy is high, so the model is good." Predicting "no" for everyone is also accurate on a rare-positive problem. The number is about the class balance.
- "The threshold is 0.5 because that is where the probability says it is more likely than not." The decision has a budget and two costs. 0.5 is the threshold for a symmetric cost with no budget, which is no real decision.
- "AUC improved, ship it." AUC describes the whole ranking. The retention team calls the top two hundred; a model that reorders users at rank ten thousand can raise AUC without changing one call.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a probability needs a threshold to become a decision, and that the threshold is set by costs and budget rather than by 0.5, holds for every classifier and every domain.
- TASK-SPECIFICThe fixed-budget framing turns classification into a top-k ranking problem; a classification whose decision has no budget — an automatic approval — is instead thresholded on the cost ratio, and the metric is expected cost at the threshold rather than precision at k (Thresholding).
- CONTESTEDA serious position holds that churn prediction is the wrong target and uplift modelling — predicting who changes behaviour when called — should replace it, because calling sure-cancellers and sure-stayers both waste the budget. The counter is that uplift needs a randomised intervention history that most teams do not have, and a churn model plus a randomised holdout is the way to build that history.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Product analytics — designing the randomised holdout and reading its lift is an experimentation question; this lesson assumes an experimentation practice exists that can say whether two hundred calls changed anything.