Bucketing & Normalisation
Bucket edges, means and standard deviations are fitted on the training fold and shipped with the model. Refit them anywhere else and the model receives inputs from a transformation it never learned.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Scaling and bucketing look like cleaning. Why are they part of the model, and what goes wrong when serving recomputes them?
An ad platform predicts click probability with a model that takes bid price, user age, and dozens of counts on very different scales. Someone standardised the inputs. The model was good offline; after a marketing campaign brought in a wave of younger users, its calibration drifted badly and the bidding system overspent for a week.
Put every feature on the same scale so no single one dominates, and keep the scaling current in serving so the inputs stay in range as traffic changes. Scaling is preprocessing, not modelling.
When the campaign lowered the mean age of traffic, the serving scaler's running mean followed. The same 22-year-old who was "young" — a negative standardised value — in training became "average" — near zero — in serving. The model, which had learned that young users click more, stopped seeing anyone as young.
- When the campaign lowered the mean age of traffic, the serving scaler's running mean followed. The same 22-year-old who was "young" — a negative standardised value — in training became "average" — near zero — in serving. The model, which had learned that young users click more, stopped seeing anyone as young.
- The running statistics also reacted to every traffic burst, so the model's inputs shifted with the time of day even though the population had not changed. Calibration wobbled continuously and nobody could say why.
- A bucketed feature — age quantiles — was recomputed from serving traffic too, so the bucket edges moved with the campaign and the "top quintile" bucket meant a different age range every hour.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict the probability that an impression is clicked. The label is a click within the attribution window.
- The output feeds a bid, so calibration matters as much as ranking: a probability that is systematically high overspends.
- One example is one impression with numeric features spanning orders of magnitude — a bid in cents, an age in years, counts in the thousands.
- The serving service standardised each feature using a running mean and standard deviation over the last hour of traffic, because that was easy and "kept the inputs in range".
- Training standardised on the training set's statistics, which were never written down.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Standardisation is
(x - mu) / sigmawithmuandsigmafitted on the training fold. The model learns weights against that specific map. Apply a differentmuandsigmain serving and the model receives a different input for the same raw value; every learned weight is now applied to the wrong quantity. The map is part of the model. - Bucketing — quantile bins, fixed-width bins, log bins — is the same: the edges are fitted state. Trees are indifferent to monotone rescaling, which is why scaling is often skipped for tree models, but bucket edges change which raw values share a leaf, and a tree learned against one set of edges is wrong under another.
- Whether scaling is needed depends on the model. Gradient-descent-trained models — linear, logistic, neural — need inputs on comparable scales for the optimiser to converge sensibly and for regularisation to penalise fairly (Regularisation). Tree ensembles do not care about scale at all. Distance-based models such as k-NN depend on it entirely.
- The population shifting is real and is a drift question (Feature Drift). Letting the scaler follow the shift hides the drift from the model and from the monitor at once: the inputs stay "in range" while their meaning changes.
The map is part of the model
The model never saw an age. It saw (age - mu_age) / sigma_age with the training values of mu and sigma, and it learned a weight against that. A serving path that computes its own mu from the last hour has changed the input without changing the raw value. When the campaign lowered the mean age, a 22-year-old moved from clearly young to unremarkable, and the model's "young users click more" weight had nothing to act on.
This is why the transform belongs in the artifact. The statistics are learned parameters; they were fitted on the training fold and they are as fixed as the weights. Serving loads them and applies them. It never computes them.
The service keeps a rolling mean and standard deviation per feature over recent traffic and standardises each request against them, so inputs always look "in range".
The scaler is fitted on the training fold, serialised with the model, and applied unchanged. Out-of-range inputs are clipped or flagged, and the raw distribution is monitored separately.
The weights were learned against one map. A moving map changes the meaning of every input and hides population drift from the model and the monitor simultaneously; a fixed map keeps the model's inputs meaningful and makes drift visible as drift.
1# training: fit on the training fold only, then freeze2mu = X_train.mean(axis=0)3sigma = X_train.std(axis=0) + 1e-94lo, hi = np.percentile(X_train, [0.5, 99.5], axis=0) # clipping bounds5artifact["scaler"] = {"mu": mu, "sigma": sigma, "lo": lo, "hi": hi}6 7def transform(x, scaler):8 # serving applies the stored map; it never recomputes a statistic9 clipped = np.clip(x, scaler["lo"], scaler["hi"])10 out_of_range = (x < scaler["lo"]) | (x > scaler["hi"]) # monitored, not hidden11 return (clipped - scaler["mu"]) / scaler["sigma"], out_of_rangeThe out_of_range flag is the monitoring hook. When the campaign changes the population, this rate rises — which is the drift signal a running scaler would have suppressed.
Buckets have edges, and edges are fitted
Bucketing turns a continuous value into a category: which quintile of price, which band of age. It helps a linear model capture non-monotone effects and makes a feature robust to outliers. The edges are computed from data, which means they are fitted state with the same rules as a scaler: fixed on the training fold, shipped, never recomputed.
Quantile bins are the tempting case, because "top 20% of prices" sounds like a stable concept. It is not; it is a set of edges that depends on the population the quantiles were computed on. A serving path that recomputes quintiles from live traffic has moved every edge, and the model's learned effect for "bucket 5" now applies to a different range of raw values.
Every scaling statistic, bucket edge and clipping bound applied in serving is the one fitted on the training fold and stored in the artifact.
holds when The fitted transform is serialised inside the artifact; serving loads and applies it; a deploy-time contract test compares its parameters to the training run's log; no serving component computes a statistic.
breaks when A serving refactor introduces running normalisation "for robustness"; a bucketing library in the serving path uses a different quantile definition; the artifact is updated without the transform; scaling is applied twice.
respond Restore the frozen transform. Then, separately, read the out-of-range rate and raw-distribution drift to decide whether the population shift warrants retraining — a decision, not a reflex (Drift Is Not Failure).
Which models need it
Scaling is not universal. A gradient-descent optimiser on unscaled inputs takes tiny steps along the large-scale features and huge ones along the small-scale ones, and L2 regularisation penalises the small-scale features' large weights unfairly. Distance-based models measure distance in the raw units, so an unscaled count in the thousands swamps an age in years. Tree ensembles split on thresholds and are indifferent to any monotone rescaling.
So the decision is by model family, and the serving-equivalence rule applies to whatever is chosen. A tree model that skips scaling still has bucket edges, clipping bounds and imputation values that were fitted and must match.
Which numeric preprocessing does this model family need, and what fitted state does it create?
when Always scale; log-transform heavy tails first; consider bucketing to capture non-monotone effects in a linear model.
cost Fitted mean, spread, log offset and bucket edges must ship with the model; regularisation strength is now scale-dependent and must be tuned after scaling.
when Skip scaling; bucketing rarely helps; clip or leave outliers since splits are threshold-based.
cost Fewer fitted parameters, but imputation values and any bucket edges still must match; teams wrongly conclude "no preprocessing to ship".
when Scale every dimension so distance means something; choose the scaling with the distance metric in mind.
cost The model's notion of similarity is entirely determined by the fitted scaling; a serving mismatch changes which neighbours are found, not just a score.
How to build it
Most important first.
- Fit
mu,sigma, bucket edges and any clipping bounds on the training fold only, store them in the artifact, and apply them unchanged in serving (Preprocessing Lives in the Artifact). Serving never computes a statistic. - Choose the transform for the model family: scale for gradient-descent and distance-based models; skip it for tree ensembles; log-transform heavy-tailed counts before scaling so the standardised value is not dominated by outliers.
- Clip to the training range or flag values outside it, so an out-of-range input is visible as such rather than silently extrapolated.
- Monitor raw feature distributions at the serving boundary, before the fixed transform, so a population shift shows up as drift instead of being absorbed by a moving scaler.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Equality of the artifact's stored statistics with the training run's logged statistics, as a deploy-time check — the number that says the map shipped intact.
- Calibration on production outcomes by segment, once labels arrive; a moving scaler shows as calibration that wanders with traffic composition.
- The rate of serving inputs outside the training range per feature, which a fixed transform makes visible and a running one hides.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every scaling statistic, bucket edge and clipping bound in the serving path is the one fitted on the training fold and stored in the artifact, and no serving component computes a statistic from traffic.
- The raw feature distributions at serving are close enough to training that the fixed transform maps most inputs into the range the model learned, and the out-of-range rate is monitored.
- The transform is applied exactly once, in the same order relative to other steps, in both paths.
- Offline: a contract test asserting the artifact's preprocessing statistics equal the training run's logged values.
- Offline: replay production requests through both paths and diff the transformed vectors; a systematic offset in one feature is a scaler mismatch.
- Online: monitor raw distributions and out-of-range rates per feature; monitor calibration by segment as labels arrive.
What can go wrong
- The statistics are frozen in the artifact, and a genuine population shift makes a large fraction of inputs fall outside the training range; the model extrapolates and the fixed scaler is blamed for what is actually drift needing a retraining decision.
- The scaler is frozen but the bucket edges are recomputed by a different library in the serving path, with a different quantile definition, so the same raw value lands in a different bin.
- A feature is scaled in training and not in serving, or scaled twice, because the pipeline object and the serving code each "handle scaling".
- Frozen statistics mean a real population shift produces out-of-range inputs and a visible drift alert instead of being quietly absorbed — which is the right behaviour and more operational noise.
- Bundling the transform into the artifact couples serving to the training framework's serialisation format.
- Log transforms and clipping are more decisions to record and to reproduce, and each is a place the two paths can differ.
- "Keep the scaler current so the inputs stay in range." Keeping inputs in range by moving the map changes what every input means. The model learned one map; it needs that map.
- "Trees do not need scaling, so preprocessing is irrelevant for tree models." Scaling, yes. Bucket edges, clipping, log transforms and imputation are still fitted state that must match.
- "The inputs drifted, so the scaler should adapt." If the inputs drifted, the question is whether the model should be retrained, and a frozen transform is what makes that drift visible. An adaptive scaler answers the question by hiding it.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- MODEL-SPECIFICGradient-descent and distance-based models need scaled inputs; tree ensembles are invariant to monotone rescaling and skip it. Bucket edges, clipping and log transforms are fitted state for every family.
- GENERALThat a fitted transform is part of the model and must be applied unchanged in serving holds regardless of model family or modality — it is the same rule as for a tokeniser vocabulary or an image normalisation constant.
Where the depth lives
This domain teaches the model and hands the rest off by name.