The Fraud Model Erased Its Own Labels
Decide what you would do from the brief alone, including whether you would change anything at all. Everything below it is available, but the exercise stops working if you open it first.
Eighteen months after launch, the fraud model's measured precision is excellent and its recall is "unknown but probably fine". The retrain pipeline reports each new version as better than the last. Chargebacks have been rising for two quarters. Risk leadership wants to know whether the model has degraded and, if so, why the metrics did not show it.
Retraining on the last three months with a higher class weight on fraud, because chargebacks rose. The new model catches more of the recent fraud pattern — the one the previous model missed — and its validation numbers improve. It is trained on an even more censored window, so it has seen even fewer of the older patterns, and the fraudsters who were using those patterns two years ago find them working again. Chargebacks fall for a quarter and rise the next, and the retrain is repeated. The loop is now running with a higher gain.
Read this even if you are confident. It is here rather than behind a button because it is the answer most teams actually ship, it passes review, and its cost arrives weeks later when the labels do.