Feature and Model Versioning
A feature name is a contract whose definition changes; a model artifact is a file whose meaning depends on which definition it was trained against. The two versions must travel together, and a mismatch is a production failure with no error message.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
How are feature definitions and model artifacts versioned so that the model in production is always fed the features it was trained on?
The feature team improved days_since_last_login to use the event stream instead of the nightly snapshot — more accurate, and it ships to the feature service on Tuesday. The churn model, trained in February on the nightly version, keeps running. By Thursday the retention team asks why the model is calling everyone a churn risk.
Features have names; the model asks for features by name; the feature service computes the current definition. Improvements to a feature benefit every model that uses it automatically.
The improved days_since_last_login has a different distribution — the event-stream version counts hours, the snapshot version counted whole days with a lag — and the model's learned weight for it now multiplies a number from a distribution it never saw. Every score shifts.
- The improved
days_since_last_loginhas a different distribution — the event-stream version counts hours, the snapshot version counted whole days with a lag — and the model's learned weight for it now multiplies a number from a distribution it never saw. Every score shifts. - No error anywhere. The feature name matched, the type matched, the service returned a value. The only signal is the retention team asking a question two days later (Feature Drift).
- Retraining fixes the model on the new definition and breaks it again on the next "improvement". The fix that was applied is a treadmill, not a control.
- Rolling back the feature change breaks a second model that was trained last week on the new definition. Two models, one feature name, two definitions, and the service can only serve one.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The surrounding model predicts churn; the versioning target is that the artifact in production is only ever fed feature values computed by the feature-definition version it was trained on, and that this can be checked mechanically at deploy time and at request time.
- The label of "model version" has to mean the whole tuple — weights plus preprocessing plus the feature-definition version — not just the weights file.
- A feature repository: named features, each with a definition (source, window, aggregation, null policy) that evolves. A feature service that computes them at request time from the current definitions (Feature Stores).
- A model registry: artifacts with weights, a preprocessing step and metadata. The metadata has a training date and a dataset version and, until this incident, nothing about features beyond their names (The Model Registry).
- Request logs showing the feature values the model received, which after Tuesday come from a different distribution under the same name (Train / Serve Skew).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A feature definition is code: a source, a transformation, a window, a null policy. Changing any of these changes the values the name produces, and a model's weights are a function of the value distribution it was trained on. So a definition change is a change to the model's input contract, and it has to be versioned like one (Preprocessing Lives in the Artifact).
- Versioning a feature means the definition has an immutable version identifier — a content hash of the definition, or an explicit version number that is never reused — and the feature service can compute any version that a live model needs, not only the latest.
- Versioning a model means the artifact's metadata names the exact feature-definition versions it was trained against, alongside the dataset version and the preprocessing. The deploy step checks that the serving side can provide those versions; the request path checks that it did (Serving Contract Tests).
The four versions travel together
Dataset v12, Labels v3, Feature Pipeline v7, Model v19: a production model is the tuple, and the artifact's metadata has to name every element. The dataset and labels say what it learned from; the feature version says what its inputs mean; the model version says which weights and preprocessing.
The deploy check reads the tuple and asks the serving side whether it can honour it. The request path stamps what it actually used. Between them, a mismatch becomes an alert on Tuesday rather than a question on Thursday.
1model: churn-30d2version: 193artifact_digest: sha256:9c1f…e24trained_at: 2026-02-14T03:12:00Z5run_id: run-4826inputs:7 dataset_version: ds-v12 # content hash recorded in run-4828 label_version: labels-v39 feature_definitions: # immutable ids; the serving side must compute these10 days_since_last_login: feat-dsll-v2 # nightly snapshot, whole days11 sessions_7d: feat-s7d-v412 plan_tier: feat-plan-v113preprocessing:14 bundled: true # scaler + encoders fitted on ds-v12, inside the artifact15reproducibility: statistical # band recorded in run-48216serving:17 reject_on_feature_version_mismatch: trueThe line that would have prevented the incident is days_since_last_login: feat-dsll-v2. On Tuesday the feature service moved to v3; a deploy or request-time check against this metadata fails loudly instead of serving a plausible number.
A mismatch is a skew with a version number
The feature change produced train/serve skew: the model applies weights learned on one distribution to values from another. Versioning does not remove the skew — it makes it detectable at the boundary, by comparing two identifiers, before any value is scored.
That is the difference between a monitor and a control. A distribution monitor sees the shift a day later; a version check sees the mismatch on the first request.
No offline change: the model was not retrained, its validation metrics on the February snapshot are exactly as recorded.
From Tuesday, scores shifted upward for most users and the retention team's call list tripled; the feature service reported healthy throughout.
- 1The feature under the same name switched from a nightly whole-day snapshot to an hourly event-stream value with a different scale and no lag; the model's weight for it was learned on the old scale.
- 2Nothing versioned the feature definition, so no deploy or request check existed to notice the artifact's contract was broken.
- 3The feature change shipped on the feature team's cadence, decoupled from the model's; there was no mechanism that made the two teams' releases visible to each other.
What must stay true for the versions to protect anything
The versions are identifiers; the protection comes from the checks that read them. An artifact that names its feature versions but is deployed by a pipeline that does not compare them is documented, not protected.
The assumption to state is about the serving side: it must be able to compute what the artifact names, and it must say what it actually computed.
For every request, each feature value the model receives was computed by the definition version the artifact names, and the response records which versions were used.
holds when Feature definitions are immutable per version; the service serves multiple versions; deploy verifies availability of every named version; requests are stamped and mismatches alert or reject.
breaks when A version is retired while a model references it; a definition is edited under its id; a fallback path computes features from a different source without a stamp; an upstream source changes meaning under an unchanged definition.
respond Stop serving from the mismatched version — fall back or reject — and either restore the named version or retrain and re-promote against the new one; never patch the metadata to match the service.
When a feature definition changes, what happens to the models that use it?
when Never for a model in production; acceptable only in exploration with no artifact promoted.
cost Silent skew on every dependent model; the Tuesday incident.
when Few models, one team, similar cadences.
cost The feature team waits on the slowest model; the coupling gets bypassed under pressure.
when A shared feature platform with many models on different cadences.
cost Old versions to compute and maintain; a reconciliation job to know when a version can retire.
How to build it
Most important first.
- Give every feature definition an immutable version and treat a definition change as a new version, never as an edit; the old version stays computable for as long as a live model references it.
- Record the feature-version set in the model artifact's metadata at training time, from the feature pipeline that produced the training matrix, so the artifact carries its contract (What a Model Artifact Contains).
- At deploy, verify that the feature service can serve every version the artifact names; at request time, stamp the response with the versions actually used and reject or fall back on mismatch (Serving Fallbacks).
- Version the preprocessing with the artifact — encoders, scalers, vocabularies — as part of the same bundle, so "model v19" is weights plus everything between raw features and tensor.
- Migrate models to a new feature version deliberately: retrain against it, evaluate, promote through the registry, and only then retire the old definition.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The count of requests per day where the feature versions served differ from the versions the artifact names. This should be zero and it is the number that would have caught Tuesday.
- The number of live models referencing each feature-definition version, which tells the feature team what they can retire and what a change would break.
- Do not measure "the feature service is healthy". It was healthy on Tuesday.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The feature service computes, for every feature the artifact names, the exact definition version the artifact was trained on — and continues to for as long as the artifact is live.
- The artifact's recorded feature-version set was captured from the pipeline that actually produced its training matrix, not assembled by hand.
- A definition version is immutable: its source, window, aggregation and null policy cannot change under the same identifier, and its upstream inputs still mean what they meant.
- Offline: a contract test that loads the artifact, reads its feature-version set, and asserts the feature service can compute each; a replay of logged requests through the artifact's versions to confirm the values match training within tolerance.
- Online: a version stamp on every prediction — feature versions used, artifact digest — with an alert on any mismatch between served and expected (Prediction Logging).
- Over time: a monthly reconciliation of registry entries against feature-service versions, flagging every live model whose versions are scheduled for retirement.
What can go wrong
- Feature versions are recorded but the version is a git branch name that keeps moving.
- The service supports multiple versions but the cost of computing old versions grows, and someone retires a version that a canary model still references (Canary Rollout).
- The artifact names feature versions but the preprocessing inside it was fitted on a different training matrix — an encoder vocabulary from a previous run — and the mismatch is inside the bundle.
- A feature's definition is unchanged but its upstream source changed meaning — a status code was repurposed — and the version says everything is fine (Semantic Changes).
- A feature service that computes multiple versions of a feature carries more code and more compute, and the old versions are the ones nobody wants to maintain.
- A strict mismatch policy means a feature upgrade cannot benefit a model until it is retrained, which is the correct behaviour and feels like bureaucracy to the feature team.
- Capturing the full contract in the artifact makes the artifact larger and the training pipeline more coupled to the feature platform.
- "The feature got more accurate, so every model using it got better." The model learned weights for the old distribution. A more accurate feature from a different distribution is a different feature, and the model applies the wrong weights to it.
- "Version the model and you are done." The model version names a weights file. Without the feature-definition versions in its metadata it does not say what inputs the weights were for, and the deploy check has nothing to check.
- "Retrain on the new feature and the problem is solved." Until the next change. Retraining is the migration; versioning is the control that makes the migration deliberate rather than an incident.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALAny model whose inputs are computed by code that can change independently of the model has this coupling; it is sharpest with a shared feature service and many models, and present even with one model and one batch job.
- SCALE-SPECIFICWith one model and one team, the feature code and the model can be released together and versioning is a git tag; with a shared feature platform and dozens of models on different retraining cadences, per-feature versioning and multi-version serving are the only way a feature change can be safe.
- CONTESTEDA serious position holds that multi-version feature serving is expensive and that the cleaner control is coupling: a feature change forces retraining and re-promotion of every dependent model before it ships, so there is only ever one definition live. The counter is that with many models on different cadences that coupling blocks the feature team on the slowest model, and in practice the coupling is bypassed — which is exactly how Tuesday happens.
Where the depth lives
This domain teaches the model and hands the rest off by name.