The Retraining System That Promotes Itself
Decide what you would do from the brief alone, including whether you would change anything at all. Everything below it is available, but the exercise stops working if you open it first.
A weekly retraining system retrains the churn model, evaluates it on a held-out slice of the latest data, and promotes it to production automatically if the AUC beats the current champion. It has promoted forty-one times in a year. The retention team reports that the model's flagged list has become "erratic week to week" and that campaign performance has drifted down.
Adding a rule that the candidate must beat the champion by at least 0.005 AUC. It is a one-line change to the gate, it will cut the promotion rate, and it looks like the fix for noise. The threshold is arbitrary — it may be inside or outside the actual noise depending on the week's holdout size — the evaluation is still on immature labels, calibration and slices are still unchecked, and a model that clears 0.005 by flagging an entirely different population still promotes. Fewer coin flips are being taken; each one is the same coin.
Read this even if you are confident. It is here rather than behind a button because it is the answer most teams actually ship, it passes review, and its cost arrives weeks later when the labels do.