The Scores Became Prices
Decide what you would do from the brief alone, including whether you would change anything at all. Everything below it is available, but the exercise stops working if you open it first.
A lending product multiplies the default model's score by the loan amount to compute an expected loss, which sets the interest rate. Finance reports that realised losses are running at roughly half of expected losses across the book (illustrative), and pricing is losing deals to competitors.
Retraining without upsampling and shipping the new model because "that was the cause". It was one cause. A gradient-boosted classifier trained on the natural class balance is still not calibrated by default — it is typically overconfident at the extremes — and nobody has drawn the reliability diagram yet. The new model's expected losses move closer to realised, which looks like a fix, and pricing is now wrong by a smaller amount in a direction nobody has measured. The step that was skipped — plotting the diagram — is the only one that would have shown whether the fix was complete.
Read this even if you are confident. It is here rather than behind a button because it is the answer most teams actually ship, it passes review, and its cost arrives weeks later when the labels do.