Regularized Linear Models
L1 makes weights zero, L2 makes them small, elastic net does both — and all of them penalise a feature in proportion to its scale, so the scaler is part of the model.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The linear model has hundreds of features and unstable coefficients between retrains. What does a penalty on the weights do, which penalty, and why does it only mean something after scaling?
A credit team's application-scoring model has grown to a few hundred features, many of them variants of each other. Every retrain reshuffles which of the near-duplicate features carry the weight, the reason codes given to declined applicants change from month to month, and the compliance team has stopped trusting them.
Fit the unregularised model on everything. More features cannot hurt a linear model; it will set the useless ones to small weights. If the coefficients bounce between retrains, that is noise in the data, not a modelling problem.
Collinear features let the fit split one effect across several weights in arbitrary proportions. Tiny changes in the training set move the split, so the 30-day spend weight is large and positive one month and the 90-day one carries it the next. The predictions barely change; the reason codes flip.
- Collinear features let the fit split one effect across several weights in arbitrary proportions. Tiny changes in the training set move the split, so the 30-day spend weight is large and positive one month and the 90-day one carries it the next. The predictions barely change; the reason codes flip.
- With hundreds of features and a modest number of defaults, the unregularised model fits noise in the rare-default segments; validation on a later period shows the overfit that the random split hid (Overfitting).
- Someone adds an L2 penalty without scaling. Income, in tens of thousands, gets a tiny weight that the penalty barely touches; a ratio in [0, 1] needs a large weight to matter and is crushed. The penalty selected features by their units.
- An L1 penalty is added to "pick the important features". It keeps one of the three spend windows, essentially at random among near-duplicates, and the compliance team asks why the 60-day window is "important" this month and not last month.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether an approved applicant will default within twelve months. The label exists only for approved applicants, so the training set is the population a previous policy chose to approve (Selection Bias).
- The score drives an approve/decline decision and the model must produce stable, explainable reason codes — the coefficients are a regulated output, not a diagnostic.
- One example is one approved application: income, debt ratios, several bureau scores, account ages, and dozens of aggregates over the same underlying transactions at different windows.
- Many features are near-collinear: 30-day, 60-day and 90-day spend are correlated above 0.9, and two bureau scores measure nearly the same thing.
- Features are in wildly different units — income in tens of thousands, ratios in [0, 1], counts in single digits.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Regularisation adds a penalty on the weights to the loss: L2 adds λ·Σwᵢ², L1 adds λ·Σ|wᵢ|. The minimiser now trades fit against weight size, and λ sets the exchange rate. This is the same idea as in Regularisation applied where it is easiest to see.
- L2 shrinks every weight towards zero in proportion to its size and spreads the effect of collinear features across all of them evenly — the split that was arbitrary becomes determined, which is what makes the coefficients stable. It never sets a weight to exactly zero.
- L1's penalty has a constant gradient of ±λ regardless of weight size, so small weights are pushed all the way to zero and stay there: the solution is sparse, which is feature selection by the optimiser. Among near-duplicates it keeps one, and which one depends on noise.
- Both penalties are on the raw weight, and a weight's size depends on its feature's unit. Scaling features to unit variance makes λ mean the same thing for every feature; without it the penalty is a statement about units. Elastic net combines the two, αL1 + (1−α)L2, keeping sparsity while letting correlated features share the weight.
Two penalties, two different geometries
Add λ·Σw² to the loss and the gradient gains a term 2λw: every weight is pulled back towards zero in proportion to its size. Large weights are pulled hard, small ones gently, none reach zero. Add λ·Σ|w| instead and the pull is a constant λ in whichever direction moves the weight towards zero — small weights get pushed all the way there and the loss gradient has to fight to keep them.
For a group of collinear features the two behave completely differently. L2 minimises Σw² subject to the combined effect being right, which spreads the weight evenly across the group — deterministic, stable, and every member of the group becomes a reason code. L1 minimises Σ|w| for the same combined effect, which is achieved by putting all of it on one member — sparse, and the member chosen is whichever the noise favoured.
1import numpy as np2 3def penalised_loss(w, base_loss, lam, alpha):4 # alpha = 1 → lasso (L1); alpha = 0 → ridge (L2); between → elastic net5 l1 = np.sum(np.abs(w))6 l2 = np.sum(w * w)7 return base_loss + lam * (alpha * l1 + (1 - alpha) * l2)8 9def penalty_gradient(w, lam, alpha):10 # L1 pulls with constant magnitude lam·alpha; L2 pulls in proportion to w11 return lam * (alpha * np.sign(w) + (1 - alpha) * 2 * w)12 13# the bias b is not penalised: shrinking it would distort the base rate, see sigmoid-and-probabilityThe gradient shapes are the whole story: a constant pull zeroes small weights, a proportional pull never does. Neither knows anything about units, which is the next section.
The penalty is in the units of the feature
The model needs income in tens of thousands to have a weight around 0.00003 and a debt ratio in [0, 1] to have a weight around 3 for the two to matter equally. The L2 penalty sees 0.00003² and 3² and shrinks the ratio's weight a hundred million times harder. Nothing about the ratio's predictive value was consulted. After standardising both to unit variance, weights of equal importance have equal size and the penalty treats them equally.
So the scaler is not preprocessing; it is part of the model's definition. Fit it on the training fold only, ship it in the artifact, and never re-fit it in the serving path.
looks like scaler.fit(X) on the whole table, then a train/validation split, then the regularised fit. A tidy notebook; a slightly-too-good validation score.
why it leaks The means and standard deviations were computed with the validation rows included, so validation examples influenced the transformation applied to training — a small leak of the validation distribution into the fit.
fix Fit the scaler on the training fold inside the cross-validation loop, as one step of a pipeline the artifact carries (Preprocessing Leakage).
Choosing between them for this model
The credit team's complaint is coefficient instability, and the fix is not "add regularisation" but "add the regularisation whose geometry matches the goal". Stable reason codes across correlated features point at L2 or elastic net with a large L2 share; a compact model with few features points at L1 after the correlated groups have been collapsed by hand.
Elastic net is the pragmatic middle: it can zero out genuinely useless features while sharing weight across a correlated group instead of picking one at random. Its second hyperparameter, the L1 share, is one more thing to defend on the validation curve.
Every served feature is standardised with the exact means and standard deviations fitted on the training fold, before the weights are applied.
holds when The scaler is serialised inside the artifact and the serving path has no scaler of its own.
breaks when The serving team "refreshes" the scaler on live traffic; a feature's upstream unit changes; a new feature is appended without a scaler entry and defaults to raw.
respond Restore the training scaler and re-run the contract test. This is a skew bug, not a modelling problem; retraining on top of it bakes the skew in.
| Option | Quality | Interpretability | Operational | Data needed | Note |
|---|---|---|---|---|---|
| No penalty | Coefficients arbitrary among collinear features; overfits rare segments; nothing to tune. | ||||
| L2 (ridge) | Stable, spread coefficients; every correlated feature becomes a reason; one λ to tune. | ||||
| L1 (lasso) | Sparse and short reason list; arbitrary choice among near-duplicates unless de-duplicated first. | ||||
| Elastic net | Sparse where features are useless, shared where they are correlated; two hyperparameters. |
caveat Interpretability here means "stable and short reason codes", which is one specific meaning; the quality column assumes standardised features and a λ chosen on a later period, and every score changes if either is skipped.
How to build it
Most important first.
- Standardise every feature on the training fold and ship the scaler in the artifact; the penalty is meaningless before that step (Preprocessing Leakage, Preprocessing Lives in the Artifact).
- Choose λ by validation on a later period, not on the training data and not by the library default; report the validation curve, not just the chosen point (Hyperparameters).
- Use L2 (or elastic net with a large L2 share) when coefficients must be stable and correlated features are expected; use L1 when a small, sparse model is the goal and you have de-duplicated the near-collinear groups first.
- For reason codes, collapse correlated feature groups into one representative *before* fitting, so that sparsity selects between concepts rather than between windows of the same concept (Feature Selection).
- Version the scaler, λ and the feature list with the model; a retrain that changes any of them changes the reason codes and needs the compliance review that implies.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Log loss or AUC on a held-out later period across the λ sweep — the curve, because the shape says how sensitive the model is to the choice.
- Coefficient stability across bootstrap refits or across monthly retrains: the standard deviation of each weight. This is the number the compliance complaint corresponds to.
- Training-set loss looks relevant and is the number regularisation deliberately makes worse.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The serving path applies exactly the scaler fitted on the training fold — same means, same standard deviations — before the dot product; a contract test on a fixed application pins the score.
- The correlation structure among features stays roughly as it was; an L1 model that kept the 60-day window is only stable while the 60-day window remains the same proxy for the group.
- λ was chosen for this feature set and this data size; a feature-set change re-runs the λ sweep rather than inheriting the value.
- Offline: the λ validation curve on a later period; bootstrap coefficient stability; a check that the scaler statistics equal the training fold's and that the serving path reproduces the notebook score on a fixed set of applications.
- Online: the score distribution and the distribution of top reason codes, weekly; a sudden change in reason-code mix with no change in applicants is a scaler or feature-set skew.
- Over time: coefficient drift across retrains, plotted per feature, with the compliance team as the reviewer of any sign change.
What can go wrong
- The scaler is fitted on the full dataset including validation, so the validation metric is slightly optimistic and the scaler's statistics are not the training fold's (Preprocessing Leakage).
- The scaler in the serving path is re-fitted on production traffic "to keep it fresh", so the weights are applied to a different standardisation than they were learned on (Train / Serve Skew).
- λ is tuned once and inherited by every retrain; the feature set grows and the same λ is now far too weak for the larger model.
- Regularisation trades training fit for generalisation and stability; a heavily regularised model will underfit a genuinely strong feature, and λ has to be defended by the validation curve, not by taste.
- L2 stability comes from spreading weight across correlated features, which makes every one of them a "reason" — explanation by committee.
- L1 sparsity gives a small model and a short reason list, at the cost of arbitrary choices among near-duplicates unless the duplicates were removed first.
- "L1 tells us which features matter." L1 tells you which features the optimiser kept at this λ on this sample; among correlated features the choice is a coin flip. Importance among duplicates is not defined (Feature Importance).
- "We added regularisation, so the model is safe from overfitting." λ was chosen on the training set or by default; regularisation at the wrong strength is either no regularisation or a strong underfit.
- "Scaling does not matter for linear models — a weight just adjusts." Without a penalty that is true. With one, the penalty sees the weight and not the unit, so scaling decides what gets shrunk.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- MODEL-SPECIFICThe sparsity-versus-shrinkage distinction and the scaling dependence are properties of penalised linear models; tree ensembles are scale-invariant and regularise through depth, leaf size and shrinkage rate instead (Tree Ensembles: When and When Not).
- CONTESTEDSome practitioners hold that L1 feature selection is the wrong tool for explainability and that a stable, interpretable model should be built by domain-driven feature curation with L2 for stability — sparsity chosen by an optimiser gives arbitrary answers among correlated inputs. Others argue that for hundreds of candidate features hand curation is not feasible and L1 or elastic net with grouped de-duplication is the only scalable route. Both are right about their regime; the difference is whether someone can afford to curate.
- DOMAIN-SPECIFICIn regulated credit and insurance the coefficients themselves are an output that must be stable and defensible, which pushes towards L2 and curated features; in ad or recommendation ranking the coefficients are nobody's business and the penalty is chosen purely on held-out loss.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Convex optimisation — the L1 penalty is non-differentiable at zero and needs a proximal or coordinate-descent solver rather than plain gradient descent, which is why library solvers differ for lasso and ridge.