ROC AUC
The probability that a random positive scores above a random negative. A pure ranking metric — invariant to threshold, to prevalence, and therefore blind to the precision that prevalence destroys.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What does ROC AUC actually measure, why does it look fine on rare-positive problems where the flagged set is mostly wrong, and when is it the right number?
A hiring platform scores applicants for "likely to pass the phone screen" to help recruiters prioritise. The model report quotes a strong ROC AUC. Recruiters say the top of the list is "no better than random". The model owner cannot reconcile the two.
ROC AUC is the standard threshold-free classification metric. A high value means the model separates the classes well. Report it and ship.
ROC AUC is computed over every positive–negative pair. Thousands of easy negatives — wrong-role applicants — are ranked below every positive, and those pairs dominate the metric. The AUC is high because the model can tell a plumber from a software engineer, which recruiters could also do.
- ROC AUC is computed over every positive–negative pair. Thousands of easy negatives — wrong-role applicants — are ranked below every positive, and those pairs dominate the metric. The AUC is high because the model can tell a plumber from a software engineer, which recruiters could also do.
- The recruiters only see the top few dozen. Among those, the negatives are the hard ones, and precision at the top is what the recruiters experience. ROC AUC says nothing about it: a model can have a strong AUC and a top-fifty list that is mostly misses (PR AUC).
- Prevalence is low, so even a low false-positive *rate* corresponds to a large false-positive *count* relative to the positives. ROC AUC works in rates, which is why it is insensitive to prevalence and why the recruiters' problem is invisible to it.
- The label exists only for applicants past recruiters chose to screen, so the AUC is measured on a population that is nothing like the daily list the model now produces.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether an applicant passes the phone screen. The label is the recruiter's decision, which exists only for applicants who were screened — a small, previously-prioritised subset (Selection Bias).
- The recruiters take the top of the ranked list each day, so the consumer is a top-k cut, not a threshold: what matters is precision at the top, not the ordering of the bottom.
- One example is one application: role, experience fields, keyword matches, source channel, and a résumé embedding.
- Screen passes are a small fraction of applications, and the recruiters look at a few dozen a day out of thousands.
- Most negatives are trivially separable — wrong role, wrong country — and the model ranks them correctly with ease.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- The ROC curve plots true-positive rate (recall) against false-positive rate (FP / all negatives) across every threshold. The area under it equals the probability that a randomly chosen positive is scored above a randomly chosen negative — a statement about pairwise ordering, which is why AUC can be computed from ranks alone without a threshold.
- Both axes are rates normalised within their own class. Multiply the number of negatives by a thousand and the false-positive rate at any threshold is unchanged, so the curve and the area are unchanged. That is the precise sense in which ROC AUC is insensitive to prevalence — and precision, which divides by the flagged count, is not.
- On a rare-positive problem a false-positive rate that looks tiny is a false-positive count that swamps the positives. At a 1% prevalence, a 5% FP rate means five negatives flagged for every positive at full recall. ROC AUC reports that as a point near the top-left corner; precision reports it as one in six.
- Because AUC is invariant to any monotone transform of the scores, it also cannot see calibration: shift or stretch every score and AUC is identical. It measures the order, and only the order (Calibration).
A probability about pairs
Pick a random positive and a random negative. ROC AUC is the probability the model scores the positive higher. That definition makes two things obvious: AUC depends only on the order of the scores, and every negative counts equally — the thousands of wrong-role applicants as much as the handful of near-miss candidates. The model gets credit for every easy pair.
It can be computed directly from ranks, which is also the fastest way: sort by score, sum the ranks of the positives, and subtract what the ranks would sum to if the positives were all at the bottom. The rank sum is the Mann–Whitney U statistic, and the sort is the whole cost (Merge Sort).
1import numpy as np2 3def roc_auc(y, p):4 order = np.argsort(p) # ascending; ties ignored for clarity5 ranks = np.empty(len(p)); ranks[order] = np.arange(1, len(p) + 1)6 n_pos, n_neg = int(y.sum()), int(len(y) - y.sum())7 # sum of positive ranks minus the minimum possible sum, over all pos×neg pairs8 return (ranks[y == 1].sum() - n_pos * (n_pos + 1) / 2) / (n_pos * n_neg)9 10# multiply the negatives by 1000 (same score distribution) and this number does not move.11# precision at any threshold falls by ~1000x. That is the whole prevalence story.The denominator is n_pos × n_neg — every pair. There is no threshold, no flagged count, and nothing that changes when negatives are added with the same score distribution.
Why it looks fine when the list is bad
The ROC axes are true-positive rate and false-positive rate: each normalised by its own class size. A false-positive rate of 5% means 5% of all negatives are flagged, whatever that number is. At one positive in a hundred, 5% of the negatives is five times the number of positives, so the flagged set is mostly negatives even at full recall — a point in the top-left of the ROC plot, and a precision the recruiters would call random.
The offline/online gap in the hiring case is exactly this. AUC was measured over all pairs and was high; recruiters measured precision at the top of a list drawn from a low-prevalence population and found it poor. Both numbers are correct. One of them is about the product.
ROC AUC on the held-out set was strong, and improved over the previous model.
Recruiters report that the top of the daily list is no better than their own skim; most of the surfaced candidates do not pass the screen.
- 1The AUC is dominated by easy negative pairs; precision at the top of the list depends on the hard negatives, where the model is much weaker.
- 2Prevalence among all applicants is far lower than among the recruiter-selected training population, so the same false-positive rate produces a far worse precision.
- 3The label came from recruiter-selected applicants, so the model was trained and evaluated on a population that does not resemble the daily list.
Where ROC AUC is the right number
None of this makes ROC AUC a bad metric. It is a very good answer to "does this model rank better than that one, independent of where we cut" — which is a model-selection question, not a product question. Its prevalence invariance means it can be compared across periods and populations with different base rates, which PR AUC cannot. And its rank-only nature means it is robust to a miscalibrated score, which is a virtue when calibration is handled elsewhere.
The decision is about the consumer. A threshold-free comparison between models wants ROC AUC. A top-k list on rare positives wants PR AUC and precision at k. A probability used in an expected-value decision wants a reliability curve, and no ranking metric will do (Calibration).
AUC and precision at k are computed on applicants drawn the way the daily list is drawn, not on the subset a previous process chose to screen.
holds when A random-sample screen provides labels on unselected applicants and evaluation uses it.
breaks when The random sample is dropped for cost; the model's own list becomes the only labelled population, so evaluation is on what the model already liked.
respond Reinstate the random sample, even small; without it every metric — ROC AUC included — is measured on a population the model curated (Feedback Loops).
What does the consumer of the score do with it, and how rare are the positives?
when Comparing models' ranking quality with no fixed operating point, or across populations with different prevalence; positives not rare.
cost Blind to precision under imbalance and to calibration; dominated by easy negatives.
when Rare positives, or a consumer that takes the top of the ranking; the precision the consumer sees is what matters.
cost Prevalence-dependent, so not comparable across periods without stating the base rate; noisier with few positives.
when The consumer takes exactly k items — a daily list, a review queue with fixed capacity.
cost Depends on k; says nothing about the ranking below k or about the model on a different capacity.
when The score is read as a probability or fed into an expected-value calculation.
cost Says nothing about ranking quality; a perfectly calibrated constant predictor has a flat reliability curve.
How to build it
Most important first.
- Use ROC AUC for what it measures: comparing the ranking quality of two models on the same data, or tracking ranking quality over time, when the operating point is unknown or will change. It is a fine model-selection number for that.
- When the consumer is a top-k cut or the positives are rare, report precision at k and PR AUC alongside it, on the population the list is drawn from (PR AUC).
- When the consumer reads the score as a probability, add a reliability curve; AUC cannot see calibration (Calibration).
- Evaluate on the population the model is applied to — all applicants, not only those past recruiters screened — which may need a random-sample screen to obtain labels (Selection Bias).
- Show recruiters precision at the top of their list, weekly, because that is the number they are already computing in their heads.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Precision at the top-k the recruiters actually look at, on a random-sample screen — the recruiters' complaint as a number.
- PR AUC, as the ranking summary that tracks the positive class.
- ROC AUC, for model comparison over time; it is the number that stays high while the top of the list is bad.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The evaluation population is the served population — all applicants — or a random sample of it, not the subset a past ranking selected; a comparison of feature distributions between screened and unscreened applicants checks this.
- The consumer's cut stays at the top-k that precision was measured on; a change in how many the recruiters review changes the number they experience.
- ROC AUC is being used as a ranking-comparison number and is not the number on which a threshold or a business claim rests; a report template that shows AUC without precision at k is a drift in usage.
- Offline: ROC AUC and PR AUC on a random-sample screen of applicants, with precision at the recruiters' k; a comparison against the previous model on the same sample.
- Online: precision at the top of the daily list as recruiters screen it; the size of the daily list.
- Over time: PR AUC and precision at k monthly; ROC AUC alongside them, so a divergence between the two is visible and explained by prevalence or the hard negatives.
What can go wrong
- Two models are compared on ROC AUC and the one that ranks the easy negatives slightly better wins, while the other had better precision at the top; the wrong model ships.
- AUC is tracked as a monitor and stays flat while calibration drifts, because AUC cannot see calibration.
- The random-sample screen that gives honest labels is dropped as wasteful, and evaluation reverts to the recruiter-selected population.
- A random-sample screen for honest labels costs recruiter time on applicants the model would not have surfaced; it is the only way to evaluate on the right population.
- PR AUC and precision at k are prevalence-dependent, so they are not comparable across roles or periods with different pass rates without stating the prevalence.
- ROC AUC is stable and comparable and that is exactly the property that makes it uninformative about the top of the list.
- "AUC is high, so the model is good." AUC is high, so the model orders a random positive above a random negative most of the time. Most of those negatives are the easy ones. Ask about precision at the top.
- "If offline AUC improved, ship it." An AUC improvement can come entirely from the bottom of the ranking, where nobody looks. Check PR AUC and precision at k, and then decide.
- "AUC did not change, so calibration is fine." AUC is invariant to any monotone transform. Calibration can be arbitrarily wrong at the same AUC.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe rank-probability interpretation and the prevalence invariance are mathematical properties of the ROC curve and hold for any scored classifier on any data.
- TASK-SPECIFICFor a balanced problem with an unknown operating point ROC AUC is a good single summary; for rare positives consumed as a top-k list, precision at k and PR AUC are the numbers that track the product, and for a probability consumer a reliability curve is needed as well.
- SIMULATEDAny AUC or precision figures in this lesson, and the 1% / 5% rate example, are for the shape of the argument; they come from the module's threshold model, not from a hiring dataset.
Where the depth lives
This domain teaches the model and hands the rest off by name.