Residuals & Assumptions
The residual plot is the diagnostic; the single metric is the summary. Heteroscedasticity, extrapolation and unscaled coefficients are all visible there and invisible in the RMSE.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The regression metric is acceptable and the coefficients look sensible. What does a residual plot show that the metric cannot, and which of the model's assumptions does it check?
A property platform predicts a listing's sale price so agents can set an asking price. Agents in one city say the model is "fine for flats and useless for houses", and a new region launched last month is getting numbers nobody trusts. The overall error metric has not moved.
Fit the linear model, report RMSE on a held-out set, check that the coefficients have the expected sign — area positive, distance to station negative — and ship. The metric is stable release to release, so the model is stable.
The error is not constant across the prediction. Cheap flats are predicted within a few thousand; expensive houses are predicted within a few hundred thousand. The RMSE averages both into a number that describes neither, and the agents' "fine for flats, useless for houses" is exactly this heteroscedasticity.
- The error is not constant across the prediction. Cheap flats are predicted within a few thousand; expensive houses are predicted within a few hundred thousand. The RMSE averages both into a number that describes neither, and the agents' "fine for flats, useless for houses" is exactly this heteroscedasticity.
- In the new region every feature is outside the training range. The line extrapolates without hesitation and the predictions are confidently wrong, but they are a tiny fraction of volume so the aggregate metric is unmoved.
- The partner-feed listings have area in square feet, so the model sees a flat "ten times larger" than it is. Their residuals are enormous and systematically positive, and they hide in the RMSE as a slightly fatter tail.
- The coefficient on floor area is "small" and on distance-to-station is "large", so a product manager concludes location matters more than size. Area is in square metres (hundreds) and distance in kilometres (units); the coefficients are in different currencies and cannot be compared.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict the eventual sale price of a listing at the time it is listed. The label is the price on the completed transaction, which arrives weeks to months later and only for listings that sold.
- The prediction sets an asking price, so being wrong by 5% on a flat and by 5% on a mansion are very different amounts of money — the cost is relative, and the spread of the error matters as much as its centre.
- One example is one sold listing: floor area, bedrooms, property type, postcode-level median income, distance to the nearest station, listing month, and photographs' count.
- Floor area is in square metres for most listings and square feet for those imported from a partner feed, with no unit column. The importer was supposed to convert.
- The new region has house prices well above anything in the training range because it is a different market.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A residual is y − ŷ for one example. Least squares guarantees the residuals sum to zero and are uncorrelated with each feature *on the training set*; it guarantees nothing about their spread, their shape, or their behaviour in any region of feature space.
- Plotting residual against prediction turns the model's assumptions into shapes. A funnel that widens with ŷ is heteroscedasticity: the error grows with the prediction, so a single RMSE is a lie about both ends. A curve is a missing non-linearity. A cluster off to one side is a subpopulation the model treats wrongly — the square-foot listings.
- Extrapolation is not an error the model can report. ŷ = w·x + b is defined for every x; the training range is a fact about the data, not about the function, and has to be stored separately and checked separately.
- A coefficient's magnitude is per unit of its feature. Standardising the features to unit variance makes coefficients "per standard deviation in this data", which is comparable across features; leaving them raw makes magnitude a statement about the unit, not the effect.
What the plot shows and the metric hides
Put predicted price on the x-axis and residual on the y-axis. A healthy fit is a horizontal band of constant width centred on zero. The property model is a funnel: narrow on the left where flats live, wide on the right where houses live. The RMSE is the width of that funnel averaged over its length, which is a number that describes no actual listing.
The same plot answers the agents' complaint directly and suggests the fix: the error is proportional to the price, so the model should predict in a space where proportional errors are constant — log price — or the loss should be relative. A second cluster floating above the band, all from one data source, is the square-foot listings, and no metric would have named them.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Residual band widens as ŷ grows | Small predictions are precise, large ones wildly off; users at the top complain | Heteroscedasticity — the error scales with the target and the loss weighted every unit equally | Model log(y) or use a relative loss; report error per prediction decile |
| Residuals curve — negative in the middle, positive at both ends | Mid-range predictions consistently too high | A non-linear relationship the straight line cannot follow | Add a transformed or bucketed feature; check whether a tree fits the shape |
| A separate cluster of residuals, all one sign | One data source or segment is always wrong the same way | A unit, encoding or null-policy difference in that source | Fix the pipeline; do not let the model learn the source as a feature |
| Residuals trend with listing date | Recent predictions biased low | The market moved; the model's intercept is stale | Time-based validation; a retraining decision, not a reflex (Time-Based Split) |
Extrapolation is silent
The new region's listings have floor areas and incomes beyond anything in training. The model produces a price for each, because a line has a value everywhere. There is no exception, no warning and no metric, because the labels will not arrive for months and the volume is too small to move the aggregate when they do.
The only defence is to record what "inside the training data" means and to check it at the boundary. For a linear model this is a per-feature range plus, ideally, a check that the *combination* is plausible — a 400 m² flat is inside every marginal range and outside the data.
Served feature vectors fall inside, or close to, the region of feature space the training set covered.
holds when The served population is the trained population — same regions, same property types, same sources.
breaks when A new region or product launches; a partner feed changes unit; the market moves the whole price distribution upward.
respond Widen the interval or decline to predict for out-of-range requests; collect labelled data from the new region before retraining, because retraining on the old data does not add the region.
1import numpy as np2 3def fit_ranges(X_train, margin=0.05):4 lo, hi = X_train.min(axis=0), X_train.max(axis=0)5 span = hi - lo6 return {"lo": lo - margin * span, "hi": hi + margin * span}7 8def out_of_range(x, ranges):9 # returns the names of features the model has never seen values like10 below = x < ranges["lo"]11 above = x > ranges["hi"]12 return np.flatnonzero(below | above)13 14# at serving time:15# bad = out_of_range(x, artifact.ranges)16# if bad.size: emit metric "extrapolation", widen the interval or fall backThe range is fitted on the training fold and shipped inside the artifact, like a scaler. Use a robust min and max (percentiles) if the training set contains outliers, or a single bad row makes every request "in range".
Coefficients on unscaled features are not comparable
The area coefficient is small and the station-distance coefficient is large only because area is measured in hundreds of square metres and distance in single kilometres. Multiply each coefficient by its feature's standard deviation and the ranking flips. Standardising before the fit produces that scaled coefficient directly.
Even standardised, a coefficient is an effect *in the presence of the other features*, in this dataset. It is a good tool for "is the sign plausible" and "which features carry the prediction" and a bad tool for "what should a seller change" (Feature Importance, Attribution Is Not Causality).
w_area = 900 per m², w_station = −40,000 per km. "Distance matters forty times more than size."
w_area = +58,000 per s.d. of area, w_station = −24,000 per s.d. of distance. "A typical spread in area moves the price more than a typical spread in distance, in this data."
A coefficient is per unit of its feature. Scaling to unit variance puts every feature in the same unit — one standard deviation of this dataset — so magnitudes compare. The numbers are illustrative; the sign flip in the conclusion is the point.
How to build it
Most important first.
- Make the residual plot part of the evaluation report: residuals against prediction, against each major feature, and against time. A metric is a summary of that plot, and the plot is where the decision gets made.
- When the error scales with the target, model log(price) or use a relative loss so that a 5% miss costs the same on a flat and a mansion (MAPE and Its Caveats).
- Store the training range of every feature in the artifact and treat an out-of-range request as a different product decision: a wider interval, a "we cannot estimate yet" state, or a fallback rule (Serving Fallbacks).
- Standardise before fitting if anyone will read the coefficients, and say so in the report; the standardiser is part of the artifact (Preprocessing Lives in the Artifact).
- Slice the metric by property type and region so the aggregate cannot hide a segment that is wrong every time (Evaluation Slices).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The residual plot and the spread of residuals per prediction decile. This is the check for heteroscedasticity, and it is the number the agents' complaint corresponds to.
- Relative error (or error in log space) per property type and per region, with the new region reported separately even though it is small.
- The fraction of requests with any feature outside the training range. This is a serving-time signal that needs no labels.
- Aggregate RMSE is the number that stayed stable while everything above went wrong.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The spread of the error is roughly constant across the prediction range, or has been made so by the transform — a monitor on residual spread per decile should stay flat.
- Requests stay inside, or near, the per-feature training range recorded with the artifact; the out-of-range counter stays near zero.
- Every feature is in the unit the model was fitted on, for every data source; a distribution check per source would show a source in different units as a shifted cluster.
- Offline: residual-vs-prediction and residual-vs-feature plots on the held-out set; relative error per property type and region; coefficients on standardised features reviewed for plausibility.
- Online: the out-of-range request rate per feature from day one; residual spread per decile as sale prices arrive, compared against the offline plot.
- Over time: repeat the residual plots on each month's production predictions once labels are in; a plot that changes shape is drift the aggregate metric will not report (Performance Decay).
What can go wrong
- The log transform fixes the funnel and a stakeholder reads the coefficients as additive amounts of money again; in log space they are multiplicative.
- The out-of-range check is set from the training set's min and max, which a single mislabelled listing has stretched to absurd values, so nothing is ever out of range.
- The residual plot is produced once at model creation and never again; six months later the region mix has moved and the plot that would show it does not exist.
- Residual plots are read by a person, so they do not fit in a CI gate the way a metric threshold does; the compromise is a few scalar summaries of the plot (spread per decile, curvature test) that can fail a build.
- Modelling log(price) makes the errors well-behaved and makes the model's output something that has to be exponentiated back, with a bias correction most people forget.
- Declining to predict outside the training range is safe and loses the launch of every new region until data arrives.
- "The metric is stable, so the model is stable." The metric is an average; two subpopulations can move in opposite directions and leave it untouched. The plot is the check.
- "Distance to station has a bigger coefficient than area, so location matters more." The coefficients are per kilometre and per square metre; standardise first or the comparison is about units.
- "It is a linear model; it will extrapolate sensibly." It will extrapolate *linearly*, which is a specific claim about a market the model never saw and has no way to verify.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALResidual diagnostics apply to any regressor — a tree ensemble has residuals and can be heteroscedastic too; only the coefficient-reading points are specific to linear models.
- DOMAIN-SPECIFICRelative error is the right frame where the cost scales with the target (prices, revenue); where the cost is absolute (minutes late, degrees of temperature) the funnel is a real property of the problem and a log transform hides it rather than fixing it.
- SIMPLIFIEDAny percentage errors or price magnitudes in the text are illustrative; the shapes in the residual plot are the lesson, not the values.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Statistics — formal tests for heteroscedasticity and for functional-form misspecification exist; this domain uses the plot because the plot also shows the square-foot cluster, which no test was looking for.