Why not always use a 0.5 threshold?
Answer it out loud before you open anything. The value of the flags below is in comparing them to what you actually said — including whether you asked about the data before naming a model.
A subscription business has a churn model that outputs a probability per customer each week. The retention team calls every customer above 0.5 with a discount offer. They have a budget for 2,000 calls a week, the model flags 9,000 customers, and about one in five of the customers called would have churned anyway (illustrative). The data scientist's answer to "why 0.5" was "that is the default".
React to this
Say what you would question, what you would trust, and what you would need to know first.
Weekly churn scoring — retention ops (illustrative)
customers scored: 310,000
base churn rate (30d): 3.1%
flagged at score >= 0.5: 9,140
call capacity: 2,000 / week
observed precision on called: 0.21 (1 in 5 would have churned)
Score histogram (share of customers):
[0.0-0.1) ######################################## 71%
[0.1-0.3) ############### 19%
[0.3-0.5) #### 7%
[0.5-0.7) ## 2.4%
[0.7-1.0] # 0.6%
Calibration: not measured. Model: gradient-boosted trees, class_weight="balanced".What it is really testing
Whether the candidate understands that a threshold is a *decision*, not a property of the model, and that it is set from costs, capacity, prevalence and calibration. The scenario contains all four: a call budget, a cost per call, a low churn rate and a score that may or may not be a probability.