Regularisation
Every way of refusing part of the training fit: L1, L2, dropout, depth limits, shrinkage, early stopping. The strength is a hyperparameter, it is tuned on validation, and for the penalty forms the features must be on one scale.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The model has more freedom than the data can constrain. How do you take some of it away without taking away the part that generalises, and how do you know how much to take?
A marketing team has a propensity model with several hundred features — engineered aggregates, one-hot campaign ids, interaction terms. It overfits: training lift is superb, validation lift is mediocre, and the coefficients are enormous and change sign between retrains. They want a model whose coefficients they can read and that does not reverse itself monthly.
Fit the logistic regression on all the features. If some coefficients look unstable, that is the data speaking; the model is finding what matters.
With correlated features the loss surface is nearly flat along the direction that trades one feature's weight against its twin's. The optimiser lands anywhere on that ridge, so the coefficients are huge, opposite in sign, and land somewhere else on the next retrain. The predictions barely change; the explanation reverses.
- With correlated features the loss surface is nearly flat along the direction that trades one feature's weight against its twin's. The optimiser lands anywhere on that ridge, so the coefficients are huge, opposite in sign, and land somewhere else on the next retrain. The predictions barely change; the explanation reverses.
- One-hot campaign ids with five rows each are fitted exactly — the model memorises those five outcomes — and the campaign feature's coefficient is a statement about five customers.
- The offline number is not terrible, so the instability is not caught until a stakeholder notices that the "top segment" flipped between the March and April reports.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict the probability a customer responds to an offer in the next thirty days. The label is a purchase attributed to the campaign within the window, which the attribution system assigns with its own errors.
- The decision is who to include in the next send; the model's ranking at a fixed budget is what matters, and stable coefficients are a product requirement because the team explains segments to stakeholders.
- One example is one customer-campaign pair. A few hundred thousand rows, several hundred features, many of them near-duplicates of each other — spend over seven days, fourteen days, thirty days — and one-hot campaign ids with a handful of rows each.
- Features are on wildly different scales: counts in the units, spend in the thousands, ratios between zero and one. This matters for what follows.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A regulariser adds a term to the training objective that penalises complexity:
loss(w) + λ · penalty(w). L2 penalises the sum of squared weights, which shrinks every weight toward zero smoothly and spreads weight across correlated features rather than letting one absorb it all. L1 penalises the sum of absolute weights, which has a corner at zero and drives weights exactly to zero — feature selection as a side effect of the objective. - Both penalties are stated in the units of the weights, and a weight's size depends on its feature's scale. A feature in thousands gets a tiny weight and is barely penalised; the same signal in units gets a large weight and is crushed. So L1 and L2 require standardised features, or the penalty regularises the units rather than the model (Bucketing & Normalisation).
- The other forms are the same idea in different clothing. Dropout randomly zeroes activations during training so no unit can rely on a specific co-adapted partner. Depth and leaf-size limits stop a tree short of one leaf per point. Shrinkage in boosting scales each new tree down so the ensemble moves slowly. Early stopping ends optimisation before the noise is fitted. Each removes a region of the function space the data cannot justify (Early Stopping).
The objective, with the penalty written out
Regularisation is not a trick applied after training; it is a change to what "best" means. The unpenalised objective says the best weights are the ones with the lowest training loss. The penalised objective says the best weights are the ones with the lowest training loss *among those that are not too large*, and λ says how strictly "too large" is enforced.
Written out, the two penalties differ in one exponent, and that exponent is the whole difference in behaviour. Squared weights have zero gradient at zero, so L2 shrinks a weight less and less as it gets small and never quite reaches zero. Absolute weights have a constant gradient right up to zero, so L1 pushes a small weight all the way there.
1import numpy as np2 3def sigmoid(z):4 return 1.0 / (1.0 + np.exp(-z))5 6def penalised_loss(w, X, y, lam, kind="l2"):7 # X must be standardised on the TRAINING fold: the penalty is in weight8 # units, and weight units are 1 / feature units.9 p = sigmoid(X @ w)10 nll = -np.mean(y * np.log(p + 1e-12) + (1 - y) * np.log(1 - p + 1e-12))11 if kind == "l2":12 return nll + lam * np.sum(w ** 2)13 return nll + lam * np.sum(np.abs(w)) # l114 15def gradient(w, X, y, lam, kind="l2"):16 p = sigmoid(X @ w)17 g = X.T @ (p - y) / len(y)18 if kind == "l2":19 return g + 2 * lam * w # shrinks proportionally20 return g + lam * np.sign(w) # constant push toward zeroCompare the two last lines. The L2 gradient vanishes as w approaches zero, so a weight is nudged but never eliminated. The L1 gradient is ±lam regardless of how small w is, so a weight whose data gradient is weaker than lam is driven to exactly zero and stays there.
Why the units are part of the model
Consider two features carrying the same information, one in cents and one in dollars. Their true weights differ by a factor of a hundred. An L2 penalty on the raw weights punishes the dollar weight ten thousand times harder for the same effect, so the model prefers the cents feature — not because it is more predictive but because its weight is numerically smaller. The penalty is regularising the choice of units.
Standardising every feature to unit variance puts every weight on the same footing, so λ means the same thing for each. The standardisation constants then belong to the artifact: they were fitted on the training fold and must be applied, unchanged, at serving time, or the penalty that was tuned is not the model being served.
looks like A scaler.fit(X) call on the full table, followed by the train/validation split. The code reads naturally and the numbers come out slightly better.
why it leaks The mean and variance used to scale the training rows were computed with validation rows included. The training fit has seen a summary of the validation distribution, and the chosen λ is tuned against it.
fix Fit the scaler on the training fold only, inside the cross-validation loop; freeze its constants into the artifact; apply them to validation, test and production without refitting.
One family of knobs
The propensity model's problem and a gradient-boosted model's problem look different and are the same: freedom the data cannot constrain. The boosted model does not have weights to penalise, so its regularisers are structural — a depth limit is a bound on how many interactions a tree can express, a minimum leaf size is a bound on how few points can set a prediction, a learning rate is a bound on how far each round can move.
Every one of these is a hyperparameter with the same protocol: sweep on validation, choose the minimum, re-sweep when the data changes. The names differ per family; the discipline does not.
The regularisation strength chosen on validation remains at or near the validation optimum for the data the model is currently retrained on.
holds when The training-set size, the feature set and the feature scales are close to what they were when λ was swept, and the scaler constants in the artifact match production statistics.
breaks when The dataset grows or shrinks substantially, features are added, or the serving-side feature statistics drift away from the standardisation constants — any of which moves the optimum without moving the setting.
respond Move λ and re-fit the scaler on the new training fold. Do not keep the constants because "they worked" — they worked for the data that produced them.
| Regulariser | Model family | What it constrains | Needs scaled features? | Strength knob |
|---|---|---|---|---|
| L2 (ridge) | Linear, logistic, network weights | Weight magnitude; spreads across correlated features | Yes | λ, swept over orders of magnitude |
| L1 (lasso) | Linear, logistic | Weight magnitude; drives some weights to exactly zero | Yes | λ; sparsity rises with it |
| Dropout | Networks | Co-adaptation between units | No | Drop probability |
| Max depth / min leaf | Trees, forests, boosting | Interactions per tree; points per prediction | No | Depth, leaf size |
| Learning-rate shrinkage | Boosting | How far each round moves the ensemble | No | η, with rounds chosen by early stopping |
| Early stopping | Anything trained iteratively | How long optimisation runs | No | Patience, on validation loss |
How to build it
Most important first.
- Standardise features on the training fold, then apply the same statistics everywhere else, before any penalised fit (Preprocessing Leakage).
- Treat λ as a hyperparameter and choose it on validation with a sweep across several orders of magnitude; the right value is problem-specific and unguessable (Hyperparameters).
- Choose the form by the goal. L2 for stable, readable coefficients on correlated features; L1 when a sparse model is worth the instability of *which* twin gets kept; elastic net when you want some of both (Regularized Linear Models).
- For trees and boosting, use the structural knobs — depth, minimum leaf size, learning rate, number of rounds — in the same way: a sweep on validation, not defaults (Gradient Boosting).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Validation lift at a fixed send budget across the λ sweep. That is the number the decision uses; training lift only shows how much fit is being refused.
- Coefficient stability across bootstrap refits or across monthly retrains, if readable coefficients are a requirement — a product metric, not a model metric.
- Do not measure "how many features L1 removed" as if fewer were better. It removed one of each correlated pair essentially at random.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The feature scaling statistics baked into the artifact match the ones the serving path applies, or the penalty that was tuned is not the penalty being served.
- The correlation structure between features is stable; if two twin features diverge in production, an L2 model that split weight across them now responds to something it never saw.
- The regularisation strength remains appropriate for the training-set size, which means the sweep is re-run when the data grows, not just the fit.
- Offline: the λ sweep with training and validation curves; confirm the chosen value sits at the validation minimum and that the coefficients at that value are stable across folds.
- Online: after a rollout, compare the served feature statistics with the standardisation constants in the artifact. A mean or scale that moved is a penalty that no longer means what it did.
- Over time: re-sweep λ at each material change in dataset size; track the chosen value. A drifting optimum is the balance moving with the data.
What can go wrong
- Features standardised on the whole dataset before the split — the penalty is now tuned with validation-set statistics, and the validation number is slightly optimistic in a way that compounds with every retrain.
- The λ chosen on last year's data is hard-coded; the dataset doubles, the right λ is now smaller, and the model is under-regularised in the sense that it now refuses fit the data could support.
- Dropout left on at inference time in a hand-written serving path — the model serves a random sub-network and its predictions are noisy per request (Train / Serve Skew).
- Regularisation trades variance for bias. A well-regularised model refuses some real structure along with the noise, and the price shows up as a higher training error the team must learn to accept.
- L1's sparsity is convenient and its choice among correlated features is arbitrary, so the "selected features" story told to stakeholders is partly an artefact of the optimiser.
- Standardisation is one more transform that must be reproduced at serving time, and one more place where a constant can silently go stale.
- "L1 found the important features." It found *one representative* of each correlated group, chosen by numerical accident. Refit on a resample and the representative changes.
- "Use the library default for λ." The default is a number that worked on someone else's dataset. The right value differs by orders of magnitude between problems and is only findable by sweeping.
- "Regularisation is for linear models." Depth limits, leaf sizes, learning-rate shrinkage, dropout and early stopping are all regularisers; a boosted ensemble with no constraints on any of them overfits as readily as an unpenalised regression.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat an added penalty or a structural constraint trades variance for bias is a property of estimation; the mechanism transfers across linear models, trees, boosting and networks even though the knobs have different names.
- MODEL-SPECIFICThe scaling requirement is specific to the penalty forms — L1, L2 and anything else stated in weight units; tree-based regularisers (depth, leaf size) are scale-invariant because a split threshold is the same in any units.
- CONTESTEDFor heavily over-parameterised networks a strong position holds that explicit regularisation matters less than the implicit regularisation of the optimiser, the architecture and the data augmentation — that dropout and weight decay are second-order effects and the real regulariser is SGD itself. That view is well supported empirically for large models with large data; it does not transfer to a logistic regression on a few hundred thousand rows, where explicit λ is the whole game.