ArtifactsGENERALFRAMEWORK-SPECIFICSIMPLIFIED

Preprocessing Lives in the Artifact

The normaliser's means and standard deviations, the encoder's vocabulary, the imputation values, the feature order and the threshold are fitted on the training fold and ship with the weights. Recomputing any of them at serving time is a different model.

Target & data What to measure What must stay true ▶ Follow One Prediction

The problem, the obvious approach, and why it breaks

Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.

The question

The serving path has to turn a raw record into the tensor the model expects. Where do the numbers that transformation needs come from, and what happens if it computes them itself?

The problem

A demand-forecasting team ships a model to a warehouse-management service owned by another group. That group asks, reasonably, for "the preprocessing spec" so they can implement it in their language. Three months in, the forecasts are consistently high for new products and the two teams disagree about whose implementation is wrong.

The obvious approach

Preprocessing is data plumbing, and the serving team owns the plumbing. Describe the steps — standardise these, encode that, fill missing with the median — and let them implement it near the request, where the data is. The model file holds the learned part; the transformation is just code.

Why it breaks

Zero is not the training median. For a new product every lag is missing, every lag becomes zero, zero standardises to a large negative value with the training statistics, and the model reads "sales far below any product it has seen" as an input — and, through interactions the tree learned, forecasts high for a low-history item on promotion.

How it breaks — usually after the offline metric looked fine

Zero is not the training median. For a new product every lag is missing, every lag becomes zero, zero standardises to a large negative value with the training statistics, and the model reads "sales far below any product it has seen" as an input — and, through interactions the tree learned, forecasts high for a low-history item on promotion.
The serving team recomputed the standardisation statistics from a recent window "to keep them fresh". Every feature is shifted relative to what the weights expect, and the shift changes weekly.
A new category appears; the training encoder maps it to the unknown bucket, the serving encoder appends it to the vocabulary, and the one-hot vector is one element longer than the model's input layer.
Offline evaluation was perfect because it ran inside the training process with the fitted objects in memory. The mismatch has no offline signature at all (Train / Serve Skew).

Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

What is being predicted, and from what data

This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.

Target

Predict units sold per product per week for the next four weeks; the label is realised weekly sales, known at the end of each week.
The prediction drives replenishment orders, so a systematic bias for one class of product — new products — produces over-stock on exactly the items with the least history.

Data

One example is one product-week with lagged sales, price, promotion flags, category, and days since launch. Lags are missing for new products.
Training standardised the lag features using training-fold statistics, filled missing lags with the training-fold median of that lag, and encoded category with a vocabulary from the training fold, with an explicit unknown bucket.
The serving team implemented the same steps from a prose document. Their imputer fills a missing lag with zero, which is what their language's numeric default is.

How it actually works

Precisely enough to predict its behaviour — not a framework API.

Every fitted transformation has parameters learned from the training fold: μ and σ for standardisation, the vocabulary and unknown policy for an encoder, the fill value for an imputer, the bin edges for a bucketer, the target statistics for a target encoder (Bucketing & Normalisation, Categorical Encoding, Missing Data, Target Encoding). The weights were fitted against inputs produced by *those* parameters.
Recomputing them from any other data — serving traffic, a fresh window, the full dataset — changes the mapping from raw record to tensor, and the weights do not know. The fix in training is to fit on the training fold only (Preprocessing Leakage); the fix in serving is to never fit at all, only apply.
The feature order is a parameter too. A model consumes a vector by position; a manifest that lists names in order is what lets serving assemble the vector correctly from a record whose fields arrive in any order.
The threshold is the last transformation: it turns the model's probability into the decision the business evaluated. It was chosen on the validation fold under a policy (Thresholding, Threshold Selection); a serving path that applies a different one serves a different decision.

The median is a weight

A normaliser fitted on the training fold learned two numbers per feature. An imputer learned one. An encoder learned a vocabulary and a rule for what is not in it. The network learned its weights *against inputs produced by those numbers*. Ship the weights without them and the serving path has to invent them — and it will invent them from whatever is nearest, which is the current data or the language's default.

This is the same mechanism as preprocessing leakage seen from the other side. Fitting the normaliser on the full dataset leaks validation statistics into training; fitting it on serving data leaks the future into the input. The case below is the serving-side mirror: the fitted value is fine, the recomputed one is a different model.

leakagelag_1_sales (standardised)Recomputed at serving — the mirror of preprocessing leakage

looks like A correctly standardised numeric feature in both training and serving, with a mean near zero in the training fold and a plausible distribution in serving logs.

why it leaks It does not leak; it *shifts*. Serving recomputes the mean and standard deviation from a recent window, so the same raw lag maps to a different standardised value than it did in training. The weights apply training-era coefficients to serving-era numbers.

offline

None. Offline evaluation ran with the fitted normaliser in memory, so the metric measured the right transformation and was correct about it.

production

Every product's lag features are shifted by the difference between the training statistics and the window statistics; the shift changes with the window, so the forecast bias moves week to week and looks like drift.

fix Store the training-fold statistics in the artifact and apply them; never fit at serving time. If the statistics need to change, retrain the weights against the new ones and ship both under one digest.

when this feature is fine A statistic computed at serving time is legitimate when it is a *feature*, not a transformation parameter — the mean sales over the last four weeks as an input the model was trained on — because then training and serving both compute it from the entity's own history at prediction time (Point-in-Time Correctness).

What each format can carry

The question the serving team asked — "send us the spec" — has three answers with different costs. A prose spec is a second implementation waiting to diverge. An explicit data document is a second implementation with a contract: the numbers are shipped, only the application is reimplemented. An artifact with the transformations embedded in its graph is one implementation, bounded by what the runtime can express.

Framework-native serialisation can carry almost anything, because it carries code; interchange formats carry a graph and its arrays and reject what they cannot express. Neither is "the answer"; the design is to put each part of the preprocessing in the format that can hold it and verify the whole with a replay.

Weights only against pipeline plus weights

Weights only, with a prose preprocessing spec

The serving team implements standardisation, imputation and encoding from a document, computing statistics from whatever data is near the request; the feature order is inferred from the document's section order.

Pipeline + weights under one digest

Weights in a parse-only format; a preprocessing document with per-feature means, standard deviations, fill values, vocabularies with unknown policies, bin edges, the ordered feature list and the threshold; a stratified replay sample; all covered by one digest and loaded together.

The serving path applies numbers it is given rather than deriving numbers it guesses, the replay sample proves the application matches training before rollout, and a retrain cannot ship new weights with old preprocessing because the digest covers both.

Apply, never fit

The serving-side code is deliberately boring: read the fitted state, assemble the vector in manifest order, apply each transformation with stored parameters, apply the stored threshold. There is no fit anywhere in the serving path, and a code review can check that by searching for it.

The stratified replay sample is what turns "we followed the spec" into "we reproduce the training outputs", and it has to include the cases where implementations differ by default — every field missing, an unknown category, a value outside the training range.

must stay trueNothing is fitted at serving time

Every transformation parameter the serving path applies — statistic, vocabulary, fill value, bin edge, threshold — is the training-fold value shipped in the artifact, and weights and preprocessing state in production were fitted in the same run.

holds when The serving code contains no fitting, only application from the shipped document; the artifact digest covers weights and preprocessing together; the stratified replay passes in the serving implementation before rollout.

breaks when A "freshness" change recomputes statistics from a window; a language default fills a missing value; a retrain redeploys weights while a cached preprocessing document stays; a new category is appended to the vocabulary at serving time.

how you would know Transformed-feature distributions at the serving boundary compared to the training fold on rollout day; the replay diff in CI; a check that the preprocessing digest and the weights digest changed together on every deploy.

respond Restore the shipped state and re-run the replay; if the statistics genuinely need to move, retrain the weights against them and ship both under one digest.

Serving-side application of fitted state — note the absence of any fitting

1state = load_json(bundle, "preprocessing.json")   # fitted on the training fold, shipped with the weights
2 
3def to_vector(record: dict) -> list[float]:
4    out = []
5    for f in state["features"]:                     # manifest order, not dict order
6        v = record.get(f["name"])
7        if f["kind"] == "numeric":
8            if v is None:
9                v = f["fill"]                       # training-fold median, not 0 and not None
10            out.append((v - f["mean"]) / f["std"])  # training-fold statistics, never recomputed
11        elif f["kind"] == "category":
12            idx = f["vocab"].get(v, f["unknown_index"])   # unknown policy from training
13            out.extend(1.0 if i == idx else 0.0 for i in range(len(f["vocab"]) + 1))
14    return out
15 
16def decide(record: dict) -> bool:
17    p = model.predict_proba(to_vector(record))
18    return p >= state["threshold"]                  # chosen on validation under the policy, shipped here
19 
20# replay check, run in the serving image before rollout
21for row in load_parquet(bundle, "replay_sample.parquet"):   # stratified: all-missing, unknown category, out-of-range
22    assert abs(model.predict_proba(to_vector(row["record"])) - row["expected"]) < 1e-6

The fill value and the unknown index are the two lines that differ between implementations by default. The replay sample has to contain rows that exercise both, or the assertion is green for the wrong reason.

How to build it

Most important first.

Ship the fitted state with the weights as explicit data: a JSON or similar document with per-feature means, standard deviations, vocabularies, unknown policies, fill values, bin edges, the ordered feature list, and the threshold. The serving path reads it and applies it; it never computes a statistic.
Where the model runtime allows, fold the transformations into the artifact's graph so the serving path hands over the raw record and the artifact does the rest; this removes the second implementation entirely, at the cost of tying preprocessing to the runtime.
Where a second implementation is unavoidable — a different language, an edge device — treat equivalence as a tested contract with a replay sample stored in the artifact (Serving Contract Tests), stratified to include the edge cases: all-missing lags, unknown categories, values outside the training range.
Version the preprocessing state together with the weights under one digest (Artifact Integrity); a bundle whose weights and preprocessing were fitted in different runs is two half-models.

What to measure

Which number actually maps to the decision — and which numbers look relevant and are not.

The mismatch rate between the serving transformation and the training transformation on the replay sample, per feature — the number that catches a wrong fill value or a refitted statistic before rollout.
The per-feature distribution of transformed values at the serving boundary compared to the training fold's transformed distribution: if the standardised lag features have a mean far from zero in serving, a statistic was recomputed.
Not the forecast error on the training data. It was computed with the correct transformations in memory and says nothing about the serving path.

What must stay true after deployment

The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.

Assumptions

Every statistic, vocabulary, fill value, bin edge and threshold the serving path applies is the value fitted on the training fold and stored in the artifact, and no serving component recomputes any of them.
The feature vector serving assembles has the manifest's features in the manifest's order, and an unknown category or missing value follows the same policy training used.
Weights and preprocessing state deployed together were fitted in the same run; the artifact digest covers both.

How to verify — offline, online, and over time

Offline: the equivalence test over a stratified replay sample, in the serving implementation, run in CI on every candidate; a unit test that a record with every field missing produces the training-fold fill values, not language defaults.
Online: monitor the transformed feature distributions at the serving boundary against the training fold on rollout day; a shifted mean on a standardised feature is a recomputed statistic (Feature Drift).
Over time: on every retrain, assert that the preprocessing state's digest changed together with the weights' digest and that serving loaded both from the same bundle.

What can go wrong

Failure modes in production

The preprocessing document is shipped and the serving team reads it once; the next retrain changes the vocabulary and the statistics, the weights are redeployed, and the old preprocessing state stays cached — a bundle whose halves no longer match.
The graph-embedded approach works until a transformation the runtime cannot express is needed — a string operation, a lookup against a table — and it quietly moves back into the serving code without a contract test.
The replay sample is a uniform draw from validation and contains no all-missing product, so the zero-fill bug passes the equivalence test.

What the recommended approach costs

Explicit preprocessing state is a format to design and maintain, and every new transformation type needs a representation and a loader on the serving side.
Embedding transformations in the model graph removes the second implementation but restricts what preprocessing can do to what the runtime can express, and makes the artifact framework-bound.
A stratified replay sample must be constructed deliberately and updated when new edge cases appear; a stale sample gives a green test for the wrong reason.

Misreads

"Preprocessing is deterministic code, so it does not need versioning." The code is deterministic; its parameters are learned. The median that fills a missing lag is as much a learned parameter as a weight.
"Recomputing the statistics on fresh data makes the model adapt to drift." It makes the inputs drift relative to the weights. Adapting to drift means retraining the weights with the new statistics, together (Drift Is Not Failure).
"An interchange format solves this." An interchange format can carry the graph and the arrays; it cannot carry a fitted encoder with a custom unknown policy unless that policy is expressed as graph operations. What it cannot carry still has to ship as data.

Where this applies

ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

GENERALThat fitted transformation parameters are part of the learned function, and that recomputing them shifts every input relative to the weights, holds for any model that consumes a transformed vector — linear, tree, network alike.
FRAMEWORK-SPECIFICWhether preprocessing can be embedded in the artifact depends on the runtime: tabular pipeline objects bundle it in a code-bearing serialisation, graph runtimes can express arithmetic and lookups but not arbitrary string logic, and interchange formats carry what their operator set allows and nothing more.
SIMPLIFIEDThe two-team story compresses several real mismatches into one; in practice the fill-value, the recomputed statistic and the vocabulary divergence arrive at different times from different changes, and any illustrative shift mentioned here is for the shape of the argument rather than a measurement.

Where the depth lives

This domain teaches the model and hands the rest off by name.

Data Engineeringfeature-pipelines nullability-and-defaults encoding-schemes

Domains that do not exist yet

— Programming Languages & Runtime Internals — the reason a numeric default differs between the training language and the serving language is a runtime question; this domain only insists that the fitted value ship as data so the runtime default never applies.