Feature Stores
A feature store is optional infrastructure that makes one feature definition serve both training and low-latency inference, with lineage attached. It is one answer to skew, not a prerequisite for ML.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Three teams compute "customer 30-day spend" three different ways and each ships a model on their own version. When is shared feature infrastructure worth its cost, and what does it actually guarantee?
A bank's fraud, credit and marketing teams each maintain their own feature pipelines. The platform lead wants one feature store so a definition is written once and reused; the fraud lead says her team has two models and a replay test that already passes, and she does not want a migration.
Adopt a feature store. Every feature goes through it, every model reads from it, skew disappears by construction, and the bank gets reuse and lineage for free. The vendor deck says this is how mature ML organisations operate.
The migration takes three quarters, during which the fraud team's existing replay test — which was working — is deprioritised because "the store will fix it". Skew appears in the interim from an unrelated service change.
- The migration takes three quarters, during which the fraud team's existing replay test — which was working — is deprioritised because "the store will fix it". Skew appears in the interim from an unrelated service change.
- The store guarantees that the same definition is executed on both sides; it does not guarantee the values were computed at the same *time*. The offline store materialises daily and the online store updates on every event, so training sees end-of-day values and serving sees mid-day ones (Feature Freshness).
- Reuse produces coupling. The marketing team changes the window on
spend_30dto match a campaign period, and the credit model — trained on the old definition — silently receives the new one. - The online store becomes a hard dependency in the request path. Its p99 becomes the model's p99, and an outage in a shared component takes down three products instead of one (Serving Fallbacks).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The surrounding systems predict chargebacks, default within twelve months, and campaign response. Each consumes customer-level aggregates that are conceptually shared and practically divergent.
- The infrastructure question has no label. Its target is operational: the same feature name produces the same number in the training set and in the serving request, and someone can say which pipeline version produced it.
- Raw events — transactions, logins, support tickets — land in a warehouse and on a stream. Each team reads them independently and aggregates by customer with its own windows and null policies.
- Training sets are built by joining aggregates to labels in batch SQL. Serving reads aggregates from whatever each team built: a Redis cache, a nightly export to Postgres, an in-process computation.
- Lineage exists in people's heads. Nobody can say from the artifact which version of
spend_30da model was trained against.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A feature store separates a feature *definition* from its *materialisation*. The definition — entity, source, transformation, window — is registered once. An offline store materialises historical values keyed by entity and timestamp for building training sets; an online store holds the latest value per entity for low-latency reads.
- Offline/online consistency comes from executing one definition in both materialisations, not from magic. Some stores compile the definition to both a batch engine and a stream engine; others compute in batch and push results to the online store, which means the online value is exactly as fresh as the last batch run.
- The offline store is what makes Point-in-Time Correctness cheap: because every value carries the timestamp at which it became known, a training set can be assembled with an as-of join instead of a bespoke backfill per feature.
- Lineage is a by-product of registration: the artifact records the feature definition versions it consumed, so a change to a definition can be traced to every model it affects.
What a store actually guarantees
Strip the vendor language and a feature store is three things: a registry of feature definitions, an offline store of historical values keyed by entity and timestamp, and an online store of current values keyed by entity. Everything else — consistency, lineage, reuse — is a consequence of routing both training and serving through the same three things.
The guarantee is narrower than the pitch. One definition executed in two places gives you the same *computation*; whether the two places see the same *events at the same time* is a materialisation question the architecture does not answer for you.
The migration versus the test
The fraud lead's position is not laziness. Her replay test compares the batch feature path against the serving path on a stratified sample every night and blocks promotion on a mismatch. For two models that is most of what the store would buy, at a cost of one script.
What her test does not cover is timing — whether the batch values her training set used were the ones that existed at each transaction's timestamp — and what happens when a third team wants her spend_30d. Those are the honest arguments for the platform, and they scale with team count, not with model quality.
| Option | Quality | Latency | Cost | Operational | Note |
|---|---|---|---|---|---|
| Replay-equivalence test over two implementations | Catches definition skew on the sampled paths; blind to timing and to paths the sample misses; must be kept green when either side changes. | ||||
| Log serving features, train on the log | Training sees exactly what serving saw; new features need a logging change before any training data exists; a privacy review for what is stored. | ||||
| Feature store with offline and online materialisation | Definition consistency, point-in-time joins and lineage by construction; a platform to run, a migration, and a new dependency on the request path. |
caveat The scores assume a handful of models; with thirty models across five teams the store's cost column improves and the test's quality column degrades, because thirty replay tests do not stay green. The matrix cannot express that the answer moves with organisation size.
Consistency is not freshness
A store that materialises the offline table nightly and updates the online store per event has consistent definitions and inconsistent timing. Training sees spend_30d as of midnight; serving sees it as of now. For a customer who spent heavily this morning the two values differ by the morning, and the model was trained on the midnight one.
This is the assumption the store cannot enforce for you, and it is the one most teams discover after adoption rather than before.
The value the online store returns for an entity at request time is the value the offline store would record for that entity at that timestamp.
holds when Both materialisations consume the same event stream with the same watermark, or the online store is refreshed from the same batch the offline store records, so both lag by the same amount.
breaks when The online store updates per event and the offline store snapshots daily; late-arriving events are applied to the offline history but were never visible online; a backfill rewrites offline history with today's code.
respond Align materialisation cadence, or train on logged online values instead of the offline store, so the model learns the freshness it will actually be served.
How to build it
Most important first.
- Decide what problem the store is solving — skew, reuse, point-in-time joins, lineage — and check whether a smaller thing solves it. A replay-equivalence test solves skew for two models; a versioned SQL view with snapshots solves lineage for a warehouse-only team.
- If adopting a store, version feature definitions and pin the artifact to specific versions. Reuse without pinning is a shared mutable variable across teams.
- Treat the online store as a service with an SLO and a fallback. A feature read that times out must return a defined default the model was trained to see, not an exception.
- Log the feature vector the model actually received, whatever the store returned. The store's idea of the value and the model's input can diverge through a client-side cache or a serialisation bug.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Offline/online mismatch rate on a replay sample — the same measurement skew needs, now applied to the store's two materialisations. A store that passes this delivers its main promise.
- Online store read latency at p99 within the request budget, and the fraction of requests that fell back to defaults. These are the numbers the product feels.
- Feature reuse counts are the number the platform team reports; they measure adoption, not correctness, and a widely reused wrong feature is worse than three private right ones.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The online and offline materialisations of a definition produce the same value for the same entity and event history, verified by a replay test rather than assumed from the architecture.
- Every model artifact names the feature definition versions it was trained on, and a definition change cannot reach a serving model without a redeploy of that model.
- The online store's read latency and availability stay inside the model's request budget, and the fallback path returns values the model was trained to interpret.
- Offline: rebuild last month's training set from the offline store and compare it row by row against the training set the model was actually trained on. Any difference is either a backfill or a definition change.
- Online: sample production requests, recompute their features from raw events through the batch definition, and diff against what the online store returned.
- Over time: alert on feature age in the online store per feature, and on the fraction of requests served with default values.
What can go wrong
- The definition is shared but the null policy is in the client: the fraud client maps a missing value to the training median, the credit client to zero. Consistency at the store, skew at the model.
- Backfilling a new feature into the offline store uses today's code over yesterday's events, producing values that could not have been computed at the time — a store-mediated form of Temporal Leakage.
- A schema change in the source table breaks materialisation for one feature; the online store keeps serving the last known value for days, and no monitor watches feature age.
- A store is a platform: a team to run it, a migration for every existing model, and a new hard dependency on the serving path. For a small team the platform costs more than the models it protects.
- Reuse trades independence for consistency. A definition change now needs a review across every consuming model, which slows the team that owns the feature.
- Offline materialisation at a fixed cadence is cheap and reproducible but freezes freshness; per-event materialisation is fresh and expensive, and harder to make point-in-time exact.
- "Feature stores are mandatory for production ML." They are not. A team with one model, one batch pipeline and a replay test has production ML without one; the store earns its cost at some number of models, features and teams.
- "We have a feature store, so skew is solved." Definition skew is solved; timing skew, client-side null policies and serialisation bugs are not, and the store cannot see any of them.
- "The store gives us lineage." It gives lineage for features that go through it. The one computed in the request handler because it needed a value from the request body is still invisible.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- SCALE-SPECIFICBelow a handful of models owned by one team the store is overhead: a replay test and a versioned SQL view do the job. Above several teams sharing entity-level aggregates, the duplicated pipelines and untraceable definitions cost more than the platform.
- CONTESTEDThe strongest case against is that a feature store is a large, opinionated system whose main promise — offline/online consistency — can be had from a replay-equivalence test on a sample of production requests, and that most of its other promises (reuse, lineage) are organisational problems a platform does not fix. The strongest case for is that equivalence tests decay, cover only what the sample covers, and say nothing about point-in-time timing, which the offline store makes cheap by construction.
- FRAMEWORK-SPECIFICWhether a store executes one definition in two engines or computes in batch and pushes to the online store differs by product, and decides whether "consistency" includes freshness. Read the materialisation model before believing the consistency claim.
Where the depth lives
This domain teaches the model and hands the rest off by name.