Calibration
Does 0.8 mean 80%? The reliability curve answers it bin by bin. Calibration matters when the probability is multiplied by a value; it matters not at all for a pure ranking.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
When does it matter that the model's probabilities are true frequencies, how do you check, how do you fix it, and when can you ignore it?
An online lender uses a default-probability model in two places: a ranked queue for underwriters, and an expected-loss calculation that sets the interest rate. The queue works well. The rates are wrong — loss provisions have been consistently under the realised losses — and the model team points out the model's ranking metrics are excellent.
The model outputs a number between 0 and 1 called a probability. The ranking metrics are strong, so the model is good. Multiply the probability by the exposure and that is the expected loss.
The up-weighting inflated every score; the boosted model's raw outputs are additionally bunched away from 0 and 1 in a way that is not a constant shift. The queue is fine, because the order is right. The expected-loss calculation is wrong for every loan, because the value is not (Sigmoid & Probability).
- The up-weighting inflated every score; the boosted model's raw outputs are additionally bunched away from 0 and 1 in a way that is not a constant shift. The queue is fine, because the order is right. The expected-loss calculation is wrong for every loan, because the value is not (Sigmoid & Probability).
- The ranking metrics cannot see it. ROC AUC is invariant to any monotone transform of the scores; a model whose 0.8 means 30% has the same AUC as one whose 0.8 means 80% (ROC AUC).
- Loss provisions are set from the sum of predicted probabilities times exposure. If the probabilities are systematically low in the high-risk bins, provisions are under the realised losses year after year, and the discrepancy is attributed to "the economy".
- A fix is applied — the probabilities are scaled to match the overall default rate — and the average is now right while every bin is still wrong: the reliability curve was bent, not shifted, and a single scalar cannot straighten it.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict the probability that a loan defaults within twelve months. The label arrives a year after origination and only for loans that were approved (Selection Bias, Ground-Truth Delay).
- One consumer needs an ordering; the other multiplies the probability by the exposure to set a price. The second consumer needs the number to be a frequency.
- One example is one funded loan: applicant financials, bureau data, loan terms, and origination channel.
- The model is a gradient-boosted ensemble chosen for ranking quality; its raw scores were never checked as probabilities (Gradient Boosting).
- Defaults are a small minority, and the training set was built with the minority up-weighted to help the ranking.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A model is calibrated on a population if, among cases it scores p, a fraction p are positive. The reliability curve checks this: bin cases by predicted probability, plot the observed positive rate per bin against the bin's mean prediction, and look for the diagonal. Above the diagonal the model is under-confident; below, over-confident. Expected calibration error is the bin-weighted average distance from the diagonal.
- Calibration is a property of the (model, population) pair, not of the model. A model calibrated on last year's applicants is calibrated on this year's only if the base rate and the feature–default relationship held. Reweighting, resampling and prevalence shift all break it; for logistic regression as a constant shift in log-odds, for boosted trees and neural networks in a bent, non-constant way (Sigmoid & Probability).
- Post-hoc calibration fits a map from raw score to probability on a held-out set. Platt scaling fits a logistic function of the score — two parameters, fixes shifts and mild stretches. Isotonic regression fits a monotone step function — as many parameters as it needs, fixes any monotone distortion, and overfits on small sets. Both preserve the ranking exactly, so ROC AUC is unchanged and the queue is untouched.
- Whether calibration matters depends only on what the consumer does with the number. A ranking consumer uses order; any monotone transform gives the same order, so calibration is irrelevant. An expected-value consumer multiplies p by a value; every unit of miscalibration is a unit of error in the decision. The same score can be fine for one and wrong for the other, at the same time.
The reliability curve
Bin the loans by predicted probability — ten equal-width bins is the usual start — and for each bin compute the mean prediction and the realised default rate. Plot one against the other. A calibrated model lies on the diagonal. The lender's curve sits below the diagonal in the high-risk bins: the model says 0.3 where 0.5 default, so the expected loss on exactly the loans that matter most is understated.
The curve is computed on a held-out later period, because on the training period a flexible model is calibrated by construction and the plot says nothing. And it is computed per segment, because a curve that is above the diagonal for one channel and below for another can average to the diagonal and hide both.
1import numpy as np2 3def reliability(y, p, bins=10):4 edges = np.linspace(0, 1, bins + 1)5 idx = np.clip(np.digitize(p, edges) - 1, 0, bins - 1)6 rows, ece = [], 0.07 for b in range(bins):8 m = idx == b9 if not m.any():10 continue11 predicted, observed = p[m].mean(), y[m].mean()12 rows.append((predicted, observed, int(m.sum())))13 ece += m.mean() * abs(predicted - observed) # bin-weighted distance from the diagonal14 return rows, ece15 16# compute this on a period disjoint from training, and per segment.17# a model can have ECE ≈ 0 overall with two segments miscalibrated in opposite directions.The ranking never enters. A perfectly calibrated model that predicts the base rate for everyone sits exactly on the diagonal at one point, which is why calibration and ranking are separate questions with separate metrics.
Fixing it without touching the ranking
Post-hoc calibration fits a monotone map from raw score to probability on a held-out set and ships it with the model. Platt scaling fits σ(a·score + b): two parameters, robust on small sets, able to fix a shift and a gentle stretch. Isotonic regression fits the best monotone step function: as flexible as the data allows, able to fix any bend, and prone to memorising a small calibration set. Both are monotone, so the order — and every ranking metric — is unchanged.
For the lender, the curve is bent, not shifted, and there are enough matured defaults to support isotonic on the main channels. The small channels get Platt, or the pooled map, because an isotonic staircase fitted on forty defaults is worse than a two-parameter line.
| Option | Quality | Operational | Data needed | Note |
|---|---|---|---|---|
| Raw boosted scores | Best ranking, no extra artifact, and wrong as a frequency in every bin. | |||
| Platt on raw scores | Fixes the shift; leaves the bend; cheap to refit. | |||
| Isotonic on raw scores | Fixes the bend where labels allow; a refit schedule tied to label maturity. | |||
| Regularised logistic model | Near-calibrated raw output; possibly a weaker ranking for the queue. |
caveat Quality here means calibration quality for the pricing consumer, not ranking quality — the raw boosted model wins the ranking column that this matrix does not have, which is the whole reason the lender ended up with two consumers and one model.
What does the consumer do with the number, and how many labels are there?
when The consumer sorts, takes the top k, or thresholds on a rank-derived cut and never displays or multiplies the number.
cost None now; a risk later if the consumer changes and the number is read as a frequency without anyone noticing.
when Logistic regression trained on resampled or reweighted data; the curve is shifted, not bent.
cost Exact only for logistic regression; a first-order fix elsewhere.
when The curve is shifted or mildly stretched and the calibration set is small.
cost Cannot fix a bend; two parameters is not enough for a boosted model's typical distortion.
when The curve is bent and there are enough positives in the calibration set to support a step function.
cost Overfits small sets; the staircase must be refitted as labels mature and the base rate moves.
Calibration is a property of a population
The map straightens the curve on the period it was fitted on. Next year's applicants come from a different channel mix under different rates, and the curve bends again — not because the model changed, but because calibration was always a statement about a population. The monitor is the same plot on each maturing cohort: mean predicted probability against realised default rate, per segment.
That monitor also distinguishes two failures that look alike. If the curve has shifted uniformly, the base rate moved and the map needs a refit. If it has bent in a new place, the score-to-default relationship changed and the model, not the map, has drifted — which is a retraining question, approached as one (Retraining as a Decision).
Ranking metrics excellent on every validation period; the model was promoted on ROC AUC and PR AUC each time.
Loss provisions computed from predicted probabilities have fallen short of realised losses for eight consecutive quarters, most severely on the highest-risk grades.
- 1The raw boosted scores were never calibrated; the up-weighted training set inflated them non-uniformly, and the high-risk bins are the most compressed.
- 2Ranking metrics are invariant to the distortion, so every promotion gate passed while the pricing error persisted.
- 3The base rate rose over the period and the uncalibrated model had no map to refit, so the error grew.
Among loans the calibrated model scores p, a fraction close to p default — on the loans currently being priced.
holds when The channel mix, applicant population and macro conditions resemble the calibration period; the map is refitted as cohorts mature.
breaks when A new channel opens, rates change the applicant pool, a recession shifts the default rate, or the map is inherited by a retrained model whose raw score scale differs.
respond A uniform shift: refit the map on the latest matured cohort. A new bend: the model has drifted; diagnose before retraining, and refit the map afterwards on a period disjoint from the retrain.
How to build it
Most important first.
- Decide per consumer whether the probability is used as a probability. For the queue, nothing is needed. For the pricing, calibration is a requirement and is verified before launch and monitored after (Prediction vs Decision).
- Plot the reliability curve on a held-out *later* period, on the population the price is applied to, and per segment — channel, grade — because segments can be miscalibrated in opposite directions and cancel in the aggregate (Evaluation Slices).
- Fit a calibration map on a period disjoint from training: Platt when the curve is shifted or gently stretched, isotonic when it is bent and there are enough labels; ship the map inside the artifact (Preprocessing Lives in the Artifact).
- Prefer a model family whose raw scores are closer to calibrated when the probability is the product — a regularised logistic regression is often nearly calibrated out of the box — and accept the ranking cost if there is one (Which Model Should We Use?).
- Monitor mean predicted probability against realised default rate per cohort as labels mature; a widening gap is a calibration alert that arrives before the provisions do. Explore the reliability tab in the Threshold Explorer at
/ml/threshold.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The reliability curve and expected calibration error on a later period, per segment — the number that answers "does 0.8 mean 80%" and therefore "are the provisions right".
- Mean predicted probability against realised default rate per origination cohort, as a monitor.
- Log loss or Brier score as a scalar that rewards both ranking and calibration, for tracking over time.
- ROC AUC and PR AUC look relevant to the pricing problem and are invariant to the error that caused it.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The population the prices are applied to has the same base rate and the same score-to-default relationship as the period the calibration map was fitted on; per-cohort realised rates against predicted rates test this as loans mature.
- The calibration map is applied in the serving path for the pricing consumer, and only there; a contract test on fixed applicants pins the calibrated output.
- The queue consumer continues to use the order and nothing else; if it starts thresholding or displaying the percentage, it has become a probability consumer and inherits the requirement.
- Offline: reliability curve, ECE and Brier score before and after calibration, on a period disjoint from both training and the calibration fit, per segment; ROC AUC before and after to confirm the ranking is unchanged.
- Online: mean predicted probability per cohort at origination, compared with realised default rate as the cohort matures; provisions against realised losses.
- Over time: refit the calibration map on a schedule tied to label maturity, and treat a bend that reappears after refitting as a signal that the model, not the map, has drifted.
What can go wrong
- The calibration map is fitted on the training period, which leaks the base rate; it looks perfect and does nothing for deployment.
- Isotonic regression is fitted on too few defaults and produces a staircase that memorises the calibration set; it looks flawless there and is wrong in production.
- The map is fitted once and never refitted; the base rate drifts and the "calibrated" model is now calibrated to a population that no longer exists (Concept Drift).
- A calibration map costs a labelled period that then cannot be used to evaluate the calibrated model, and a refit schedule that has to track label delay.
- Isotonic fixes any monotone distortion and overfits small sets; Platt is safe and cannot fix a bend. Choosing needs a look at the curve and a count of the positives.
- A model family chosen for calibration may rank slightly worse, which the queue consumer will notice; two consumers with different needs may end up with two models.
- "The AUC is excellent, so the probabilities are fine." AUC cannot see calibration; it is invariant to any monotone transform. Plot the reliability curve.
- "We scaled the scores to match the overall default rate, so it is calibrated now." A scalar fixes a shift. If the curve is bent, the average is right and every bin is still wrong.
- "Calibration always matters." It matters when the number is used as a frequency. A ranked queue does not care, and adding a calibration step to it adds a failure mode for no benefit.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe reliability curve and the ranking-versus-expected-value distinction apply to any probabilistic classifier; only the size and shape of the raw miscalibration depend on the model family.
- MODEL-SPECIFICA regularised logistic regression trained on the true prevalence is often close to calibrated; boosted trees, random forests, SVMs and neural networks trained with modern regularisation are frequently not, and their distortions are bent rather than shifted, which is why isotonic rather than Platt is often needed for them.
- CONTESTEDOne camp holds that every deployed probability should be calibrated as a matter of hygiene, because consumers change and a number labelled "probability" will eventually be used as one; the other holds that calibration is a requirement to be added only where the decision needs it, since a calibration map is a second fitted artifact with its own drift and refit schedule, and adding it to a pure ranking consumer creates a failure mode with no benefit. Both are defensible; the difference is whether you trust that the consumer will stay a ranking consumer.
- SIMULATEDAny probability values, bin frequencies or metric figures in this lesson are for the shape of the argument, produced by the module's threshold model rather than measured on a lending book.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Decision theory — expected-value pricing is only as good as the probability it multiplies, and the reliability curve is the empirical check on the probability; the pricing model itself belongs to finance.