Business Metrics vs Model Metrics
A model metric describes the model; a business metric describes what happened when the model's output was acted on. The map between them is the operating point and the action, and a better model metric can produce a worse business outcome.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The new fraud model has a clearly better AUC, and after rollout the fraud losses did not fall. What sits between the model metric and the business one, and which of them should decide the next release?
A payments company runs a fraud model whose output feeds a manual review queue staffed by a fixed team of analysts. The data science team reports that the new model's AUC is markedly better. Finance reports that fraud losses moved very little, and the head of risk wants to know which number is lying.
AUC is the standard fraud metric. It is threshold-free, robust to imbalance, and the new model is clearly better on it. Ship the model with the same threshold, and the business metric will follow, because a better ranking of fraud must mean less fraud.
The queue has fixed capacity. The analysts review the same number of transactions per day whichever model ranks them; the business outcome depends on the precision in the top few hundred scores, not on the ranking of the whole distribution. The new model improved the ranking mostly in the middle, where nobody looks.
- The queue has fixed capacity. The analysts review the same number of transactions per day whichever model ranks them; the business outcome depends on the precision in the top few hundred scores, not on the ranking of the whole distribution. The new model improved the ranking mostly in the middle, where nobody looks.
- The threshold was kept, but the new model's score distribution is different, so the same threshold now sends a different volume to review. The queue overflowed, analysts triaged by amount instead of by score, and the model's ranking stopped being the ranking used.
- The intervention has a cost the metric never saw: a declined good transaction loses the sale and sometimes the customer. The new model catches slightly more fraud and declines noticeably more good customers, and the net is flat.
- The label is the old policy's label. Fraud the old model declined was never observed, so the new model was evaluated on a population that excludes the cases the old model was best at (Selection Bias).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a transaction will be charged back as fraud within ninety days. The label arrives late and only for transactions that were approved — a declined transaction has no chargeback and no label.
- The model's score is thresholded into a queue; the analysts approve or decline what they review in the order the queue presents it. The business outcome is fraud loss net of the value of good transactions declined, plus the analyst cost.
- One example is an approved transaction with its ninety-day chargeback outcome, joined to cardholder and merchant aggregates. Around ten million rows over a year; positive rate under one percent.
- The training set was produced under the previous model's policy: transactions it declined never got a label, and transactions it sent to review were resolved by analysts whose decisions are in the outcome. The data is a record of what the old policy allowed to happen.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A model metric is a function of scores and labels on a held-out set. AUC in particular is the probability that a random positive is scored above a random negative, computed over the entire distribution, with every pair weighted equally. It has no threshold, no queue, no cost, and no notion of what happens to a score after it is produced.
- A business metric is a function of actions and outcomes. Between the score and the action sits the operating point — the threshold or the top-k rule — and between the action and the outcome sits the intervention: the analyst's decision, the customer's reaction, the fraudster's adaptation. Each of these is a place where the model metric and the business metric can move independently.
- The map from one to the other is the product of the operating point and the intervention. Change the model and keep the threshold, and the operating point moves; keep the operating point and change queue capacity, and the intervention moves. AUC is invariant to both, which is exactly why it is a poor predictor of the business outcome and a good description of the model.
Two numbers, one model
The offline report said the new model was better, and it was: the ranking of fraud against non-fraud improved across the distribution. The business report said losses barely moved, and it was right too. The two numbers measure different things, and the assumption that one follows from the other is where the release decision went wrong.
What connects them is a chain of things that are not the model: a threshold set on the old score distribution, a queue of fixed size, analysts who decide in the order the queue presents, and a customer who leaves when a good transaction is declined. Each link can absorb a model improvement or reverse it.
AUC rose from 0.86 to 0.91 on a held-out quarter, with the improvement spread across the score distribution.
Fraud loss in the treatment arm fell by a few percent, not the eighteen percent the offline simulation projected; declined-good-transaction value rose; net effect close to zero.
- 1The queue has fixed capacity, so the business outcome depends on precision in the top few hundred scores per day. The new model's AUC gain came mostly from the middle of the ranking, which never reaches an analyst.
- 2The threshold was inherited from the old model's score distribution and sent more transactions to review than the queue could absorb; analysts fell back to triaging by amount, and the model's ranking stopped being the one used.
- 3The intervention has a cost — declined good customers — that AUC does not count, and the new model, at the inherited threshold, incurred more of it.
- 4The labels come from the old policy: fraud the old model declined was never observed, so the offline comparison excluded the cases the old model handled best.
The operating point is the map
A score becomes an action at the operating point, and the operating point is where the model metric and the business metric are joined. Change the model and the same threshold means a different volume, a different precision and a different cost. AUC is indifferent to all of it; the business is indifferent to nothing else.
So the operating point is a decision that has to be remade for every candidate model, from its own score distribution, against the queue's real capacity and the intervention's real cost. A model shipped at an inherited threshold has not been evaluated; it has been assumed.
What must stay true for the model metric to predict the business one
Precision at the queue's capacity is a reasonable stand-in for net fraud loss only while the queue, the analysts and the intervention behave as they did when the stand-in was chosen. Grow the team, and k changes; change the decline flow, and the false-positive cost changes; let fraudsters adapt, and the label distribution changes.
The model metric therefore has to be monitored against the business metric it stands in for, in production, on the delayed labels — and the day they diverge is the day the map needs redrawing, whether or not the model has changed.
The threshold, the queue capacity and the action taken on a flagged transaction in production are the ones the offline business estimate assumed, so precision at capacity continues to predict net fraud loss.
holds when The operating point is re-derived per model from its own score distribution, queue capacity is stable, and the decline flow and its cost per good customer are unchanged.
breaks when A new model ships at an inherited threshold; the analyst team grows or shrinks; the decline path changes; the label population shifts because the new policy declines different transactions than the old one.
respond Redraw the map first — re-derive the operating point, update the cost inputs, recompute the simulation — and only then ask whether the model needs replacing.
How to build it
Most important first.
- Name both metrics before training and write down the map between them: which scores become which action, at what capacity, with what cost per intervention (Decision Before Model).
- Evaluate the model metric at the operating point the business will use — precision at the queue's capacity, recall at the chosen threshold — and re-derive the threshold for every new model from its own score distribution (Threshold Selection).
- Estimate the business metric offline by simulating the action: apply the operating point to held-out scores, charge each false positive and false negative its cost, and report the total. It is an estimate under the old policy's labels, and it is the number to compare models on (Prediction vs Decision).
- Confirm online. The business metric is only measured by acting on the model in production, so a rollout is an experiment with a control, not a launch (A/B Testing Models).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Fraud loss net of declined-good-transaction value and analyst cost, on the ninety-day delayed outcomes, in the treatment arm against the control. This is the number the decision is about.
- Precision in the top-k where k is queue capacity, offline, as the model metric that best predicts the business one under a fixed queue.
- AUC as a description of the model for the data science team. It is not the release criterion, and a rise in it is not a reason to ship.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The operating point in production is the one the offline business estimate used — the same threshold or top-k rule, at the same queue capacity, with the same intervention applied to what crosses it.
- The relationship between a model's score ranking and the fraud loss it prevents is stable enough that a model metric at the operating point predicts the business metric; a fixed-capacity queue makes that relationship a top-k precision, not an AUC.
- The labels used to compute both metrics come from a population the new model will actually face; a policy change that alters what gets declined alters what gets labelled.
- Offline: compute AUC, precision at queue capacity, and the simulated net loss for both models on the same held-out period. If the rankings disagree, the metric that agrees with the business decision is the one at the operating point.
- Online: split traffic, hold the queue capacity and intervention fixed, and compare net fraud loss on the delayed labels once the label window has closed. Report the interval, not the point.
- Over time: track precision at capacity in production against its offline value. A gap opening is either drift or a change in the operating point that nobody wrote down.
What can go wrong
- The simulated business metric is computed on the old policy's labels, which do not contain the fraud the old model declined; it overstates the new model for the population it was tuned on and says nothing about the population it will see once it is the policy.
- The online experiment is run for two weeks and the chargeback label takes ninety days; the comparison is made on the early-arriving labels, which are the obvious fraud, and the models are compared on the cases where they agree (Ground-Truth Delay).
- Queue capacity is changed mid-experiment because the new model overflowed it. The treatment arm now has a different intervention as well as a different model, and the two effects cannot be separated.
- Optimising precision at the queue's capacity ties the model to today's capacity; when the team grows, the operating point moves and the model may no longer be the best one for the new k.
- Simulating the business metric offline requires a cost per false positive and false negative that finance must supply and will dispute, and the simulation is only as good as those numbers.
- Confirming online means holding a control arm that runs the old model and therefore keeps losing the fraud the new model would have caught, for the duration of the label delay.
- "AUC went up, so the model is better." The model is better at ranking the whole population. The business only acts on the top of the ranking, at a capacity AUC does not know about. Better everywhere except where it matters is not better.
- "Fraud losses did not fall, so the model is not better." The model may well be better and the operating point wrong — the threshold inherited from the old score distribution sends the wrong volume to a queue of fixed size. Fix the operating point before judging the model.
- "We should optimise the business metric directly." The business metric is delayed, noisy, and affected by things the model cannot see. Optimise a model metric at the operating point; validate against the business metric. The two are different jobs.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a model metric describes scores and a business metric describes actions, with an operating point and an intervention between them, holds for every deployed model in every domain.
- DOMAIN-SPECIFICFraud makes the gap vivid because the queue is fixed-capacity and the label is delayed and policy-dependent; in an ads ranking system the operating point is implicit in the auction and the business metric arrives in hours, so the gap is narrower and the same mistake shows up faster.
- SIMULATEDThe AUC and fraud-loss figures in this lesson are illustrative — chosen for the shape of the argument, not measured on any real payments system.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Product analytics — the design of a business metric that is sensitive enough to detect a model change and robust to everything else is an experimentation question this domain relies on but does not own.