Train / Serve Skew
The weights are identical in training and production. The features are not. A model can be exactly right about inputs it will never see again.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The offline evaluation was strong, the artifact was promoted unchanged, and production quality is poor. What is different about the inputs?
A fraud team retrained their transaction model on six months of history. Validation precision at the operating threshold improved by a third. Two weeks after rollout, the manual review queue doubled and analysts are rejecting most of what the model flags.
The model is a file. Train it, evaluate it, copy the file to the serving cluster, point the service at it. The features are "the same features" — they have the same names.
The warehouse job computes spend_last_hour over a calendar hour bucket; the serving cache computes a rolling sixty minutes. Same name, different distribution, and every transaction near the top of an hour looks unusually quiet to the model.
- The warehouse job computes
spend_last_hourover a calendar hour bucket; the serving cache computes a rolling sixty minutes. Same name, different distribution, and every transaction near the top of an hour looks unusually quiet to the model. - The batch job treats a missing merchant category as its own category; the service maps it to
0, which is a real category. The model learned that category0is low-risk. - Offline evaluation was run on warehouse features — the training distribution — so it measured a system that does not exist in production. The number was correct about the wrong thing.
- Nothing failed. No exception, no schema violation, no latency change. The only signal was analysts saying the flags were nonsense, weeks later.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a card transaction will be charged back within 90 days. The label is the chargeback event, which arrives up to three months after the transaction.
- The decision downstream is binary — send to manual review or approve — so the model's output is a probability that a threshold turns into a queue.
- One example is one transaction, joined to the cardholder's aggregate behaviour: spend in the last hour, last day, last 30 days; merchant category counts; distance from the previous transaction.
- Training features are computed in a batch SQL job over a warehouse table. Serving features are computed in a Java service from an in-memory cache of recent events.
- The two implementations were written by different people, a year apart, from a shared document that described the features in prose.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A trained model is a function of its inputs *as distributed during training*. Its weights encode how much a unit change in
spend_last_hourshould move the score, given the range and shape that column had in the training set. - Skew is any systematic difference between the feature values the model saw in training and the values it receives at serving time for the same real-world situation. It can enter through a different implementation, a different time window, a different null policy, a different unit, a different data source, or a different moment of computation.
- Because the model never sees the "true" feature, only the number handed to it, it cannot tell that the number means something else now. It applies the learned weights to a value from a distribution it never learned, and the output is confidently wrong.
Same name, different number
The two implementations agree on the name spend_last_hour and disagree on what an hour is. The batch job, written in SQL over a warehouse, truncates to the calendar hour because that is what date_trunc does and it is fast. The serving cache, written in Java against an event stream, keeps a rolling window because that is what a cache of recent events naturally gives you.
Neither is wrong. They are different features that share a name, and the model was trained on one of them. At two minutes past the hour the batch feature holds two minutes of spend and the serving feature holds sixty; at fifty-nine minutes past they nearly agree. The model learned from the first and is being asked about the second.
looks like A well-named aggregate present in both the training table and the serving request, with plausible values in both.
why it leaks It does not leak; it *skews*. Training saw a calendar-hour bucket and serving supplies a rolling window, so the same real situation produces a different number on each side. The model cannot see the definition, only the value.
fix One definition, computed once, used in both places; or an equivalence test that replays production requests through the batch path and fails on a mismatch.
The evaluation measured a system that does not exist
The offline evaluation was correct. It reported how the model performs on warehouse-computed features, and on those features it performs well. The mistake is treating that number as a statement about production, where the model receives a different feature vector for the same transaction.
This is the general shape of the offline/online gap in this domain: the offline number is a fact about the training distribution, and every difference between that distribution and the serving one — skew, drift, a feedback loop, a broken join — makes the number describe something other than what shipped.
Validation precision at the operating threshold up by roughly a third against the previous model, on warehouse-computed features with held-out labels.
Manual review queue doubled; analysts rejecting most flags; chargeback recall unknown for another ten weeks because the labels have not arrived.
- 1The serving feature path computes several aggregates over different windows and null policies than the batch path, so production inputs come from a distribution the model never saw.
- 2The threshold was chosen on the validation score distribution, which is not the production score distribution, so the queue size was never going to match.
- 3Fraud behaviour may also have shifted in the two weeks — but that would show as feature drift over time, not as a step change on rollout day.
What must stay true after the artifact ships
The weights encode a contract with the features: this column, this unit, this window, this null policy, this range. Deploying the artifact is a promise that the serving path honours that contract, and nothing in the artifact file enforces it.
So the assumption has to be made explicit and checked — at promotion, on the first day, and whenever either side of the feature pipeline changes. A model without that check is correct by coincidence.
For the same underlying events, the serving path and the training path produce the same feature vector, to a stated tolerance.
holds when Both paths execute one definition, or an equivalence test over a representative replay sample is green and covers the rare paths — unknown categories, nulls, window boundaries.
breaks when Either implementation changes independently; a new category value appears that the two null policies handle differently; the serving cache and the warehouse disagree about which events exist at request time.
respond Do not retrain. Find the diverging feature, fix the definition on one side, and only then decide whether the model needs new weights.
1def check_skew(requests, batch_features, serve_features, tol=1e-6):2 # requests: a stratified sample of last week's production requests3 mismatches = {}4 for req in requests:5 a = batch_features(req) # the training-time path, replayed6 b = serve_features(req) # the serving path, as deployed7 for name in a:8 if abs(a[name] - b.get(name, float("nan"))) > tol:9 mismatches[name] = mismatches.get(name, 0) + 110 rate = {k: v / len(requests) for k, v in mismatches.items()}11 # a feature that mismatches on more than 0.1% of requests blocks promotion12 return {k: r for k, r in rate.items() if r > 0.001}The interesting part is the sample. A uniform sample of last week never contains the merchant category that appears once a month, so stratify by the rare values of every categorical feature or the test is green for the wrong reason.
How to build it
Most important first.
- Compute the feature once and use the result in both places: log the serving-time feature vector and train on the log, or generate both from one code path (Feature Stores exist mostly to make this possible).
- When two implementations are unavoidable, treat equivalence as a tested contract: replay a sample of production requests through the batch path and assert the vectors match within tolerance (Serving Contract Tests).
- Version the feature definition with the model. An artifact that does not name the feature pipeline version it was trained against cannot be checked at deploy time (Preprocessing Lives in the Artifact).
- Monitor feature distributions at the serving boundary against the training distribution, per feature, so skew shows up as a drift alert on day one rather than as a review-queue complaint on day fourteen (Feature Drift).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Per-feature distribution distance between the logged serving vectors and the training set, on the first day of rollout. This is the number that catches skew; validation precision cannot, because it was computed on the wrong inputs.
- Precision at the operating threshold on production outcomes, once labels arrive — the number the business cares about, and the last one to move.
- Do not measure "model accuracy" on replayed warehouse features and call it production monitoring. It re-measures the training distribution.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The serving-time feature computation produces the same value as the training-time computation for the same underlying events, to within a tolerance a test can state.
- The null, default and unknown-category policies are identical in both paths, and a new category value maps to the same thing in both.
- The feature definitions the artifact was trained against are the ones deployed with it — a redeploy of the feature service does not silently move the distribution.
- Offline: replay a stratified sample of last week's production requests through the training feature path and diff the vectors. A mismatch rate above the tolerance blocks promotion.
- Online: for the first day of a rollout, compare the logged serving feature distributions against the training distributions feature by feature, before any outcome labels exist.
- Over time: when labels arrive, compare production precision at the operating threshold against validation precision. A persistent gap with no drift alert means the measurement is wrong, not the model.
What can go wrong
- The equivalence test passes on the replay sample and the skew is in a rare path the sample did not cover — a merchant category that appears once a week.
- The feature store fixes the computation but not the timing: training reads the feature as of the end of the day, serving reads it as of the request, and the values differ by the events in between (Point-in-Time Correctness).
- The monitor alerts on every feature every day because the training set was six months old and the world moved a little. The alerts are muted, and the real skew arrives silently among them.
- A single code path for features means the batch training job now depends on a service that was built for low-latency serving, or the service on a warehouse — one of them will be the wrong tool for its job.
- Logging every serving feature vector is storage and a privacy question; it is also the only way to train on what production actually saw.
- Equivalence tests are a second thing to maintain, and they fail whenever either implementation legitimately changes, which trains people to rubber-stamp them.
- "The validation metrics were fine, so the model is fine and production has a bug." Both are true. The model is fine on inputs it will never receive.
- "We should retrain on fresher data." Retraining on warehouse features rebuilds the same skew with newer weights. It fixes nothing and costs a training run.
- "Skew means we need a feature store." It means you need one computation or a tested equivalence. A feature store is one way to get that, and not a small one.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a model applies training-time weights to serving-time values follows from what a model is, so skew is possible in every system with two computation paths, whatever the model family.
- SCALE-SPECIFICAt small scale a single Python module can compute features for both training and serving and skew is nearly impossible; the problem appears once latency forces a separate serving implementation, and grows with the number of features.
- CONTESTEDA serious position holds that the cure is worse than the disease: a feature store or unified feature platform is a large piece of infrastructure with its own failure modes and a multi-quarter migration, and for a team with a handful of models an equivalence test over a replay sample gives most of the protection at a fraction of the cost. That is right for most teams; the argument for the platform is that equivalence tests decay and do not cover timing.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — an equivalence test between two implementations is a contract test, and the discipline of keeping it green when both sides legitimately change is a testing question this domain assumes rather than answers.