Importance Says It Is the Cause

Decide what you would do from the brief alone, including whether you would change anything at all. Everything below it is available, but the exercise stops working if you open it first.

The brief you were given

The head of product reads a feature-importance chart from the churn model and asks for a roadmap item: "The top feature is number of support tickets. Let us reduce support tickets by making it harder to open one, and churn will fall." The model is accurate and the chart is real.

The trap — the fix that moves the metric and is not the fix

Producing a SHAP dependence plot that shows churn probability rising with ticket count, and sending it to product as confirmation. The plot is more detailed than the bar chart and has the shape everyone expected, so it reads as stronger evidence. It is the same evidence: a description of the model's function, which was fitted to a dataset where problems cause both tickets and churn. The roadmap item ships, ticket volume falls, churn does not, and the model — now trained on a period with fewer tickets — quietly moves its weight to login frequency.

Read this even if you are confident. It is here rather than behind a button because it is the answer most teams actually ship, it passes review, and its cost arrives weeks later when the labels do.