Fraud occurs in 0.1% of transactions. Is accuracy useful?
Answer it out loud before you open anything. The value of the flags below is in comparing them to what you actually said — including whether you asked about the data before naming a model.
A payments company is evaluating a vendor fraud model. The vendor deck reports 99.85% accuracy on a held-out set. The internal team has a rule-based system that blocks transactions from a list of known bad merchants and card BINs, and nobody has ever computed its accuracy. Leadership wants to know whether the vendor model is better than what exists.
React to this
Say what you would question, what you would trust, and what you would need to know first.
Vendor evaluation deck, slide 7 (all numbers illustrative): Held-out set: 2,000,000 transactions Fraud rate: 0.10% Accuracy: 99.85% ROC AUC: 0.96 "Our model outperforms industry benchmarks." Not on the slide: the confusion matrix, the threshold, precision at any recall, the period the held-out set covers, or how the fraud labels were obtained.
What it is really testing
Whether the candidate can reason from the *cost of each mistake* to a metric, instead of reaching for a metric by name. The 0.1% prevalence is a gift: a model that predicts "legitimate" for every transaction is 99.9% accurate, which is higher than the vendor number.