Problem Formulation Lab
The questions to answer before choosing a model, each worked for the same running problem — users are cancelling subscriptions — and, more usefully, what actually goes wrong when the field is left blank. The template earns its place in that last column or not at all.
Start from the decision, not the model. What will someone do differently on Monday because the prediction exists, when must it exist to be acted on, what does each kind of mistake cost, and can the label be observed at all? Every field below is a constraint on the ones after it, so answer them in order and out loud for a problem you actually have — then open the worked answer and compare. The what happens if you skip it block stays visible on every field, because that is the argument for the field existing.
- 1Decision
What decision do we want to improve? Not "predict churn" — what will someone do differently on Monday because the prediction exists?
What happens if you skip itThe team builds a churn model, reports an illustrative AUC of 0.84, and nobody knows what to do with it. It becomes a dashboard column that is looked at and not acted on, and six months later the project is judged a failure without ever having been aimed at anything. The characteristic symptom is a model whose "users" are analysts rather than a workflow. - 2Predicted event
What event should be predicted, exactly? Define it as a query over the data, with a horizon, and name the edge cases you are including and excluding.
What happens if you skip itThe label becomes whatever column exists — usually achurnedflag written by a billing job that nobody can define. It mixes voluntary and involuntary churn, so the model learns to predict expiring cards; it has no horizon, so a cancellation two years later counts the same as one next week; and it silently changes definition when the billing system is migrated, which retrains the model on a different event under the same name. - 3Prediction time
When must the prediction be available, relative to the event and relative to the action? This fixes what "available at prediction time" means for every feature.
What happens if you skip itFeatures are computed "as of now" in the notebook, which includes activity from the days immediately before cancellation — the user stopped logging in, opened a support ticket, removed seats. The model looks superb offline because it is predicting churn from the act of churning. In production the score arrives too late to act on, and the team asks why the model that was so accurate never gives them anyone in time to call. - 4Action
What action follows the prediction, who takes it, and how much of it can they do? The action's capacity is the threshold in disguise.
What happens if you skip itThe model outputs a probability, a threshold of 0.5 is applied by default, and on the first Monday the list has 3,000 accounts on it. The team calls the top 200 by account size — the ordering they were using before — and the model has changed nothing. Or the threshold is raised until the list is manageable, without anyone checking what precision that implies. - 5Cost of a false positive
How costly is a false positive — acting as if the event will happen when it would not have? In the business's units, not the model's.
What happens if you skip itThe false positive and the false negative are priced the same, by default, by whichever metric the library printed. Accuracy or F1 is optimised, the threshold lands somewhere that reflects neither cost, and the first sign that the costs were asymmetric is a finance query about the discount budget or a CSM lead saying the list is useless. - 6Cost of a false negative
How costly is a false negative — missing the event? Include the cases where the action would not have helped anyway.
What happens if you skip itEvery missed churner is treated as equally bad, so the model is tuned for recall over a population where most positives are tiny accounts, and the team spends its 200 calls on a long tail while the enterprise cancellation that mattered was ranked 340th. The cost was never written down, so the ranking could not be built around it. - 7Label observability
Can labels be observed reliably, and when? If the answer is "eventually, partially, and after we have intervened", say so now.
What happens if you skip itThe first monitoring dashboard reports accuracy over the last week, computed on the handful of predictions old enough to have outcomes, and it swings wildly. Nobody set up a holdout, so when the retention programme is evaluated a year later it is impossible to say whether the model found churners or the calls prevented churn — the two effects were never separable, and the next model is trained on the mixture. - 8One training example
What is one training example? A user? A user at a point in time? A subscription? The answer sets the row count, the correlation between rows and the unit of every metric.
What happens if you skip itThe dataset is one row per user, built as of today, and the question of when each row's features were frozen is never asked. Some rows have features from the day before cancellation and some from months earlier; the model learns a mixture of "what churn looks like" and "what having cancelled looks like". Row counts are reported as if rows were independent, and the validation set's confidence is overstated by the number of times each account appears. - 9Features at prediction time
Which features exist at prediction time — in the serving path, at the required freshness, for the population being scored? Not which columns exist in the warehouse.
What happens if you skip itThe model is trained on every column in the warehouse, and the top feature by importance isdays_since_cancellation_request, which is null for everyone who has not cancelled. The offline number is near-perfect. The serving team then discovers that half the remaining features are computed by a daily job that does not exist in production, and the model launches with a third of its inputs imputed. - 10Acceptable baseline
What baseline would the business accept, and what would the model have to beat? Include the rule the team uses today.
What happens if you skip itThe model is compared with predicting the majority class, which it beats trivially, and is shipped. Nobody measured the existing rule, so nobody can say whether the CSM team is now better off than when they used their own judgement — and when the team says the list "feels no better", there is no number to argue with. - 11What must stay true
What must stay true after launch for this to keep working? List the assumptions the model has weights attached to, and how you would notice each one breaking.
What happens if you skip itThe model ships and is right for a while, and the assumptions break one at a time without anyone assigned to notice. A tracking change halves the login counts on iOS; the model flags every iOS account; the team stops trusting it; the holdout is dropped to "save capacity"; the retrain on the contaminated data looks better offline and is worse. Every step was reasonable and nobody had written down what had to remain true.