The Column That Knew
Decide what you would do from the brief alone, including whether you would change anything at all. Everything below it is available, but the exercise stops working if you open it first.
A model predicting which insurance claims will be litigated jumped from a middling AUC to near-perfect after joining a new table. Product wants it deployed to route claims to senior adjusters at intake. The new table is the claims-handling system's current view of every claim.
Dropping legal_flag and assigned_adjuster_level — the two columns that obviously encode the outcome — retraining, seeing the AUC fall only slightly, and calling it clean. The obvious leaks are gone and the number is still excellent, which reads as "the signal was real". reserve_amount and days_open are still snapshot values that encode the same outcome less obviously, and the model has simply moved its weight onto them. The deployment then fails at intake exactly as it would have, a month later, after a launch announcement.
Read this even if you are confident. It is here rather than behind a button because it is the answer most teams actually ship, it passes review, and its cost arrives weeks later when the labels do.