Underfitting
A model with too little capacity, or the wrong representation, misses structure that is plainly in the data. It is the honest failure — visible offline — and still the one most often fixed with the wrong tool.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Training and validation error are both poor and nearly equal. More data does nothing. What is the model unable to express, and how do you know it is the model and not the labels?
A logistics team wants to predict delivery time from distance, time of day and vehicle type. Their linear model is off by a wide margin on both training and validation, and the errors have a shape: short trips are overestimated, long urban trips underestimated, and every rush hour is missed.
Fit a linear model on the raw columns. It is fast, interpretable, and if the error is high the answer is more data or a longer training run.
A linear model can express only "each extra kilometre costs the same minutes, in every zone, at every hour". The data says otherwise, so the model settles on an average slope that is wrong for every specific case in a predictable direction.
- A linear model can express only "each extra kilometre costs the same minutes, in every zone, at every hour". The data says otherwise, so the model settles on an average slope that is wrong for every specific case in a predictable direction.
- Adding data does not help: the model is already at its best fit, and the best fit of a line to a curve is the same line whether you have ten thousand rows or a million. Training and validation error sit together, high, and stay there.
- The failure is visible offline — both numbers are bad — but the team reads high validation error as a data problem or a noise floor. Weeks go into cleaning labels that were fine.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict minutes from dispatch to delivery for a scheduled job. The label is the actual delivery timestamp minus the dispatch timestamp, both logged by the driver app.
- The prediction feeds a customer-facing time window, so the cost of being wrong is asymmetric — a late delivery costs a support call, an early one costs almost nothing — and the decision is which window to promise.
- One example is one completed delivery: straight-line distance, dispatch hour, day of week, vehicle, and the zone ids at both ends. Two years of jobs, several hundred thousand rows — data is not the constraint.
- The true relationship is nonlinear and interacting: urban distance costs more time per kilometre than motorway distance, rush hour multiplies urban time and barely touches motorway time, and vans and bicycles respond to traffic differently.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Bias, in the decomposition, is the distance between the true function and the closest function the model class can represent. If the truth is a curve and the class is lines, that distance is fixed and no amount of data reduces it. The residuals carry the shape of what is missing: structured, not random.
- Underfitting has two sources that look identical in the metric and differ in the fix. Too little capacity: the model class cannot bend where the data bends. The wrong representation: the capacity exists but the inputs are in a form that hides the structure — raw distance when the model needs distance-by-road-type, or an hour as an integer when it needs a cyclical encoding.
- The diagnostic is the residual plot and the learning curve. Residuals with visible structure against a feature mean the model is missing that structure. Two learning curves that plateau together high, early, mean more rows will not close the gap because there is no gap to close (Residuals & Assumptions).
The line that cannot bend
A linear model in distance and hour says that every kilometre costs the same minutes and every hour adds the same offset, regardless of where the trip is. Urban rush hour breaks both claims. The model's best response is a compromise slope that under-predicts urban trips and over-predicts motorway ones, and the residuals show it: positive in the city, negative on the ring road, spiking at eight and five.
No amount of training rows changes this. The model has already found the best line; the problem is that the answer is not a line. This is the bias term from the decomposition made concrete — the distance from the truth to the nearest thing the model can say.
Features: `distance_km`, `hour`, `is_van`. The model must express traffic as a constant offset per hour and road type not at all.
Features: `urban_km`, `motorway_km`, `sin(hour)`, `cos(hour)`, `urban_km * rush_hour`, `is_van * urban_km`. Still a linear model.
The model class did not change; the coordinates did. Linear in the right features expresses the interactions that were invisible in the raw ones, and the residual structure against hour and zone disappears — which is the test that the representation was the problem.
Capacity or representation: the residuals decide
Two underfit models can have identical metrics and need opposite fixes. If the residuals correlate with a feature you already have, the model cannot express that feature's effect — add capacity or a transform of it. If the residuals correlate with something you do *not* have, such as road type inferred from zone pairs, no capacity will help and the fix is a feature.
The order of operations matters. Establish the baseline; if the model barely beats it, the model is not seeing structure. Look at residuals against every feature before touching the model class. Only when residuals are flat and the errors are still high should the labels be suspected.
Training and validation error are both high and close. Which fix applies?
when Always first. If the mean predictor or a simple rule is within reach of the model, the model has not found the structure.
cost An afternoon; and it may show the model should not exist yet.
when Residuals correlate with a feature you have, in a shape the model class cannot express: interactions, cycles, thresholds.
cost New transforms to reproduce at serving time; each is a skew risk.
when Residuals show structure across many features at once and hand-crafting each transform is impractical.
cost Variance appears; regularisation, tuning and a real validation protocol become necessary.
when Residuals are flat against every feature and the baseline is far behind; the error has no shape to chase.
cost A labelling audit, and possibly the discovery that the floor is real.
1import numpy as np2 3def residual_structure(model, X, y, feature_names):4 resid = y - model.predict(X)5 report = {}6 for j, name in enumerate(feature_names):7 col = X[:, j]8 # bin the feature and look at the mean residual per bin;9 # a flat profile means the model already captured this feature10 bins = np.quantile(col, np.linspace(0, 1, 11))11 idx = np.clip(np.digitize(col, bins[1:-1]), 0, 9)12 per_bin = np.array([resid[idx == b].mean() for b in range(10)])13 report[name] = per_bin.max() - per_bin.min() # size of the missing structure14 return dict(sorted(report.items(), key=lambda kv: -kv[1]))The largest entry names the feature whose effect the model is missing most. For the delivery data it is hour, with a profile that dips twice a day — the shape of a term the model does not have, not the shape of noise.
Consistent, and consistently wrong
The small train/validation gap is the seductive part. A model whose two numbers agree looks trustworthy, and in a narrow sense it is: it will be exactly as wrong in production as it was offline. That is generalisation of the error, not of the skill.
After deployment the assumption that must hold is that the structure the model *can* express is the structure the world has. A new city, a new vehicle class or a road closure introduces a shape the representation may not carry, and the model reverts — quietly, with a small gap — to underfitting on that slice.
The features and transforms the model was given express the nonlinearities and interactions that actually determine delivery time, for every slice of traffic it serves.
holds when Residuals are flat against every feature and pair of features on the current data, and the model beats the per-slice baseline on every slice.
breaks when A new region, vehicle or time pattern has an effect shape the current transforms do not cover; or a zone remapping breaks the urban_km split and the model silently sees raw distance again.
respond Treat it as a representation problem for that slice — inspect the residual shape, add the transform — not as a reason to retrain on the same features.
How to build it
Most important first.
- Check the baseline first. If the mean predictor is nearly as good as the model, the model has not found the structure — and if a simple rule beats the model, the structure is there and the model cannot see it (Baselines Are Mandatory, The Rule Baseline).
- Plot residuals against every feature. Structure in the residuals is the missing capacity or the missing feature, and it usually names itself: the rush-hour dip is an interaction between hour and zone.
- Add capacity or representation in the direction the residuals point: an interaction term, a nonlinear transform, a tree-based model that finds interactions on its own (Feature Engineering, Decision Trees).
- Only then re-examine the labels. Label noise raises both errors too, but its residuals are unstructured, and its learning curve keeps a small gap rather than none (Label Quality).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Training error against the baseline's error. A model that barely beats the mean predictor is underfitting whatever its validation number says relative to other models (Majority Class and Mean Predictor).
- Residual structure: correlation of residuals with each feature and with pairs of features. This is the number that says what to add.
- Do not measure the train/validation gap as the quality signal here. It is small by construction, and reading small-gap as good is how underfitting gets shipped.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The features that carry the structure — zone, hour, vehicle, and their interactions — remain populated and meaningful; a zone remapping silently reverts the model to the underfit case.
- The residual structure that was fixed does not return: a new city with a different road mix is a new nonlinearity the current representation may not cover.
- The baseline comparison is re-run at each retrain, so a model that has quietly stopped beating the mean predictor on some slice is noticed.
- Offline: plot the learning curve. Two curves that converge high and flat by a fraction of the data are bias; keep adding capacity or features until they separate, then tune from there.
- Online: compare production error per slice against the baseline per slice. An underfit model loses to the baseline on exactly the slices whose structure it cannot express.
- Over time: track residual correlation with each feature as a monitored metric. Structure re-appearing in the residuals means the world grew a shape the model does not have.
What can go wrong
- The team adds capacity by switching to a large model without fixing the representation, and now has an overfit model that is still missing the road-type structure because no feature carries it.
- A cyclical encoding of the hour is added and the model improves, so the team assumes representation was the whole problem — but the residuals against zone still have structure that a linear model cannot express.
- The noise floor is real for one slice — rural deliveries with unpredictable access — and the team keeps adding capacity chasing an error that is irreducible there, overfitting everywhere else (Evaluation Slices).
- Capacity that fixes bias creates variance to manage, which brings regularisation, validation and tuning into a system that was previously one equation.
- Interaction features and transforms are representation choices that must be reproduced exactly at serving time; every one is a new place for train/serve skew to enter (Train / Serve Skew).
- Moving from a linear model to a tree ensemble loses the coefficient-level explanation the operations team was using to reason about pricing.
- "Both errors are high, so the labels are noisy." Noise gives unstructured residuals. Structured residuals are the model failing to express something the data plainly contains.
- "More data will fix it." More data reduces variance. Bias does not move with sample size; the learning curve shows it flat from early on.
- "Small train/validation gap means the model generalises well." It means the model is consistent. A model that is consistently wrong generalises its wrongness perfectly.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALA model class that cannot represent the target function has an error floor independent of data volume; that holds for every family, though the floor is much lower for flexible families like trees and networks.
- MODEL-SPECIFICThe representation form of underfitting is most acute for linear models and distance-based methods, which see only the coordinates they are given; tree ensembles find axis-aligned interactions on their own and are underfit mostly by depth limits, not by missing transforms.
- SIMULATEDThe learning-curve shape in the explorer comes from polynomial fits to a seeded synthetic function; a degree too low plateaus high regardless of sample size, which is the argument, not a delivery-time measurement.