Threshold Selection
Flag when P × cost_FN exceeds (1 − P) × cost_FP. The threshold falls out of the costs and the calibrated probability; 0.5 is what you get when the costs are equal and nobody checked.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Why not always 0.5 — and given the costs, the prevalence and a calibrated probability, where should the threshold sit?
An insurer's claims model flags suspicious claims for investigation. The investigations team can take a fixed number a week; each investigation costs a few hundred in analyst time, and a fraudulent claim paid out costs thousands. The current threshold is 0.5 because "that is the default". The head of claims asks whether it should be.
Use 0.5. The model says fraud when it is more likely than not, and investigating claims that are more likely honest than fraudulent seems wrong. If the queue is too long or too short, nudge the constant.
At 0.5 the model investigates a claim only when fraud is more likely than not. But a fraudulent payout costs ten times an investigation, so a claim with a 20% fraud probability is worth investigating in expectation and 0.5 leaves that money on the table every week.
- At 0.5 the model investigates a claim only when fraud is more likely than not. But a fraudulent payout costs ten times an investigation, so a claim with a 20% fraud probability is worth investigating in expectation and 0.5 leaves that money on the table every week.
- The prevalence among all claims is low, so few claims ever reach 0.5; the investigations team has idle capacity while fraud is paid out. The threshold that fits the capacity and the threshold that minimises expected cost are both far below 0.5.
- The team nudges the constant to fill the queue without writing down why. Next quarter the average claim size rises, the cost of a miss with it, and the "tuned" constant is stale in a way nobody can see because it was never derived.
- The probabilities are read as calibrated. If they are not — a resampled model, a drifted base rate — the derivation produces a precise threshold for a probability that does not mean what it says (Sigmoid & Probability).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a submitted claim is fraudulent. The label is the investigation outcome, which exists only for investigated claims — the uninvestigated majority have no ground truth (Selection Bias).
- The decision is investigate-or-pay, with a cost on each mistake that the business can state in currency, which is what makes the threshold derivable rather than guessed.
- One example is one claim: type, amount, claimant history, time since policy start, repair-shop identity and free-text features from the description.
- Fraud prevalence among investigated claims is far above the prevalence among all claims, because the past threshold selected them; the population prevalence is estimated from random audits.
- The model's probabilities were checked on a held-out later period and are roughly calibrated on the audited population (Calibration).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- For a claim with calibrated fraud probability P, investigating costs cost_FP with probability (1 − P) — it was honest — and paying costs cost_FN with probability P — it was fraud. Investigating is cheaper in expectation when P·cost_FN > (1 − P)·cost_FP. Solve: P > cost_FP / (cost_FP + cost_FN). That is the threshold, and it is 0.5 exactly when the costs are equal.
- With cost_FN ten times cost_FP the threshold is 1/11 ≈ 0.09: investigate anything with more than a nine percent chance of fraud. The number surprises people until they see that a nine-percent chance of a ten-times loss is worth a one-times cost.
- The derivation uses P, which is why calibration is a precondition. A model whose 0.09 means 0.09 on the served population gives the right threshold; a model whose scores are shifted gives a threshold on the wrong scale and the whole calculation is precise nonsense.
- Capacity is a second constraint. If the cost-optimal threshold flags more than the team can investigate, the real threshold is the capacity cut, and the gap between the two is the value of hiring — a number the derivation hands the business for free. Prevalence enters through P and through how many claims exceed the threshold, never as a separate knob.
Deriving the threshold from expected cost
Take one claim with calibrated fraud probability P. Investigate it and, with probability 1 − P, you paid an analyst to confirm an honest claim: cost_FP. Pay it and, with probability P, you paid a fraud: cost_FN. Investigating is the better bet when P·cost_FN > (1 − P)·cost_FP. Rearranged, investigate when P > cost_FP / (cost_FP + cost_FN).
Everything about "why not 0.5" is in that line. The threshold is 0.5 only when the costs are equal. Prevalence is inside P. Calibration is the assumption that P means what it says. And the precision/recall trade-off is what happens to the confusion matrix as the derived threshold moves away from 0.5 — lower threshold, higher recall, lower precision, and the costs say that is the right direction.
1def cost_threshold(cost_fp: float, cost_fn: float) -> float:2 # investigate when p * cost_fn > (1 - p) * cost_fp3 # ⇔ p > cost_fp / (cost_fp + cost_fn)4 return cost_fp / (cost_fp + cost_fn)5 6cost_threshold(300, 300) # 0.5 — equal costs: the "default" is a cost assumption7cost_threshold(300, 3000) # 0.09 — a miss costs 10x: investigate anything above 9%8cost_threshold(300, 30) # 0.91 — a review costs 10x a miss: only near-certain cases9 10def per_claim_threshold(cost_fp: float, amount: float) -> float:11 # when the miss costs the claim amount, the threshold is per claim12 return cost_fp / (cost_fp + amount)The function has no model in it. The model contributes P; the threshold is entirely a property of the costs. That is the sense in which the threshold is a business decision and not a model property.
What the derivation assumes
The inequality is exact for a calibrated P. It is silently wrong for a P that is shifted or bent: a model trained on a resampled set produces P values that are too high, the derived 0.09 lands on claims that are really at 2%, and the queue fills with honest claims. The Threshold Explorer's calibration tab exists to make this failure visible before it is deployed.
It also assumes the costs are the costs. A change in claim sizes, analyst rates or regulatory penalties moves the threshold without touching the model, which is why the derivation — not just the number — is stored with the config.
Illustrative counts at the derived threshold. The FP column looks alarming until the costs are attached: 610 investigations at a few hundred each against the fraud that the lower threshold caught. Expected cost, not the size of any one cell, is the comparison.
The model's output, on the claims it is applied to, matches the observed fraud frequency bin by bin.
holds when Calibration was checked on random audits from the served population, not on investigated claims, and the base rate has not moved since.
breaks when The model was trained on resampled data without correction; the claim mix or fraud rate shifts; calibration was checked on the investigated subset, which the old threshold selected.
respond Recalibrate on recent audits and re-derive; the threshold formula is unchanged, the P feeding it was wrong.
When capacity binds
If the cost-optimal threshold flags more claims than the team can investigate, the effective threshold is the capacity cut, and every claim between the two thresholds is a bet the business is declining to make. Multiply the expected saving per claim in that band by the number of claims and the derivation has produced a business case for hiring — or for a cheaper first-pass check.
That is the practical payoff of deriving the threshold instead of tuning it: the constant comes with an explanation, the explanation names the inputs, and a change in any input has an obvious consequence.
What do you know about the costs, the calibration and the capacity?
when Costs can be stated, P is calibrated on the served population, and capacity does not bind.
cost Needs random audits to keep calibration and prevalence honest, and a cost review on a schedule.
when The miss cost scales with the case — claim amount, transaction value — and the model is calibrated.
cost The decision depends on two inputs; explaining "why this claim and not that one" is harder.
when The team's capacity is fixed and binds below the cost-optimal threshold, or calibration cannot be verified.
cost Recall floats with volume; the expected cost of the claims left below the cut should be reported as the price of the constraint.
when Costs are genuinely unknown and a held-out period with outcomes exists to tune against.
cost The cost ratio is still there, hidden in the tuned constant, and cannot be re-derived when the world changes.
How to build it
Most important first.
- Get cost_FP and cost_FN from the owners in currency, record them with the model, and derive the threshold as cost_FP / (cost_FP + cost_FN); show the derivation in the Threshold Explorer at
/ml/threshold(Decision Before Model). - Check calibration on the served population before using the derivation, and recalibrate if the reliability curve is off; the derivation is only as good as P (Calibration).
- Compare the cost-optimal threshold with the capacity cut; if capacity binds, report the expected cost of the gap so the business can decide whether to fund more investigators.
- Put the threshold and its derivation in a config the claims team owns, bound to the model version, and re-derive on every model promotion and on every cost change (Thresholding).
- Where the cost of a miss scales with the claim, derive the threshold per claim: investigate when P·amount > cost_FP, which is the same inequality with cost_FN replaced by the claim amount.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Expected cost per week at the deployed threshold, on audited outcomes: FP·cost_FP + FN·cost_FN. This is the number the derivation minimises and the one the head of claims should see.
- The reliability curve on the served population, because the derivation assumes it is flat on the diagonal.
- Queue size against capacity, and the expected cost of the claims between the cost-optimal threshold and the capacity cut.
- F1 and accuracy look relevant and both encode a cost ratio that is not the insurer's.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The model's probabilities are calibrated on the population it is applied to, checked on random audits rather than on investigated claims; a reliability curve per quarter tests this.
- cost_FP and cost_FN are the current costs, and the ratio has not moved with claim sizes or analyst rates; a quarterly cost review records any change.
- The threshold applied in production is the one derived for the deployed model version; a contract test on fixed claims pins both.
- Offline: a threshold sweep on a later period with expected cost per threshold; the minimum should sit at cost_FP / (cost_FP + cost_FN) if calibration holds, and if it does not, the gap is a calibration finding.
- Online: expected cost per week from audited outcomes; queue size; the reliability curve on random audits.
- Over time: re-derive quarterly and on every promotion; track the derived threshold as a series so a move is explained by a cost change, a calibration change, or a model change.
What can go wrong
- The costs are averages and the claims are not; a per-claim threshold using the claim amount is more accurate and harder to explain to the team.
- The random audit that estimates population prevalence and checks calibration is cut to save money, and the derivation silently drifts onto a miscalibrated P.
- The threshold is derived correctly and the config store serves it to a new model version whose scores are on a different scale.
- A derived threshold is only as honest as the costs; stating them forces a conversation some organisations would rather not have, which is the point.
- Random audits to check calibration and prevalence cost investigations that could have gone to high-score claims; they are the price of knowing the threshold is right.
- A per-claim threshold is more accurate and makes the decision a function of two inputs, which is harder to audit than a constant.
- "0.5 is the natural threshold." 0.5 is the threshold when a false positive and a false negative cost the same. Ask whether they do; here they differ by an order of magnitude.
- "We should tune the threshold to maximise F1." That sets the cost ratio to one. The business has a cost ratio; use it.
- "The threshold is 0.09, so the model is barely confident when we investigate." The model is calibrated; a 9% chance of a ten-times loss is worth a one-times cost. The threshold is low because the miss is expensive, not because the model is weak.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe expected-cost inequality is decision theory and holds for any calibrated probability from any model; what changes across domains is whether the costs can be stated and whether calibration holds.
- CONTESTEDA strong position holds that deriving a threshold from costs and a calibrated probability is fragile in practice — costs are rough, calibration drifts, and a threshold tuned directly on the business metric over a held-out period, or set by capacity, is more robust than the closed form. That is fair where costs are genuinely unknown; where they can be stated, the derivation makes the assumption explicit and the "robust" alternative hides the same assumption in a tuned constant.
- SIMPLIFIEDThe ten-to-one cost ratio and the 0.09 threshold are for the shape of the argument; the inequality is exact, the numbers are not from any insurer.
Where the depth lives
This domain teaches the model and hands the rest off by name.