Comparisons
Pairs that get conflated in real metric reviews and real design docs — batch and online, precision and recall, data drift and concept drift, validation and test. Neither column wins; what decides is the problem. Each record leads with the confusion, because the confusion is the reason the record exists.
Feature store vs no feature store
The pair is argued as if the store were mandatory infrastructure or a fad, and both are wrong. A feature store solves two real problems: the same feature computed by two code paths (train/serve skew), and features joined at training time with values from the future (point-in-time incorrectness). If those are your problems, it is the tool. The strongest case for the store is a platform team with dozens of models where every online feature would otherwise be re-implemented, and every offline join would otherwise be someone's custom SQL; there, the store is cheaper than the bugs. The strongest case against is that it is a distributed system with its own freshness, availability and cost, that its online path is another hop on the latency budget, and that a batch-scored model with features written to a versioned table gets the same guarantees for free. The dishonest version of either side is the one that does not mention the team size and the number of models. The question is "do we have the two problems, and what is the cheapest thing that solves them here", and for many teams the answer is a table, a point-in-time join and a test.
When several models share features across teams, when the same feature must be served online and used offline, and when point-in-time correctness has already bitten you. The store earns its cost by making skew structurally hard.
When there is one model, or a few, on a batch pipeline where the training features are the serving features by construction — or when the team is small enough that a well-named table and a point-in-time join do the job.
| Dimension | Feature store — a shared service that defines features once and serves them offline and online, point-in-time correct | No feature store — features computed in the training pipeline and, where needed, re-computed by the serving code |
|---|---|---|
| Solves | Skew and point-in-time joins, structurally | Nothing by itself; you solve them per model |
| Right scale | Many models, shared features, online + offline | Few models, batch, one team |
| Operational cost | A service to run, with freshness and availability | Pipelines and tables you already have |
| Latency | One more hop on the online path | Whatever your serving code does |
| Point-in-time correctness | Built in, if used correctly | Your join, your responsibility |
| Failure mode | Stale features served as fresh; store outage | Two implementations of one feature diverge |
| Signal it is time | The third model re-implements the same feature online | — |
Model families compared
Linear, trees, boosting, neural and k-NN — compared without naming a winner, and with the block that says where the comparison stops being true.
No column is a winner. Each family is a set of assumptions about the data — linear separability, axis-aligned interactions, additive residuals, a representation that can be learned, locality in feature space — and the right one is the one whose assumptions your data happens to satisfy at the size you have. The reflex answer “XGBoost” is the red flag this table exists to catch: it is often right on tabular data and it is never right as a reflex, because it skips the baseline that would have told you whether anything more than a linear model was needed. The where this comparison misleads block on every row is the part worth reading.
Count independent entities, not rows — a million events from ten thousand users is ten-thousand-sized data for generalising to new users. And the neural column flips completely with transfer learning: a pretrained model fine-tuned on two thousand images beats every other column on that task.
The "boosting wins on tabular" claim is true for a tuned model on tens of thousands of clean rows with a validation set. It is not true for three hundred rows, for a problem whose signal is linear, for a regulator who wants coefficients, or for a team with no time to tune — and the gap to the linear model is often inside the error bar.
The columns are not competing on the same input. Once a pretrained network has produced an embedding, a linear model or k-NN on top of it is often within a few points of full fine-tuning — so "neural for images" usually means "a neural representation, then whichever head is cheapest".
Interpretability is not one property. A linear model with two hundred correlated features and L1 selection is harder to explain honestly than a depth-three tree; and every column's "explanation" is undermined equally by a leaked or proxy feature, which the explanation will present with confidence.
Training cost is dominated by the number of runs, not the run: a boosted model tuned over two hundred configurations costs more than a network fine-tuned once. And the k-NN column's zero is a loan repaid on every query.
The model is rarely the slow part. Feature retrieval, a network hop and JSON serialisation usually dwarf any of these numbers, so a latency budget is a question about the serving path before it is a question about the family.
A low burden on the modelling side does not remove the burden on the serving side: every column's features must be reproduced at prediction time with the same code, the same freshness and the same point-in-time semantics. Trees remove the need to engineer features, not the need to serve them.
Native handling is convenient and dangerous in equal measure: it lets a tree model learn that "missing" predicts the target, which is fine until serving produces missingness for a different reason — a timeout, a new form — and the model reads the outage as a signal.
Calibration only matters if a downstream decision multiplies the score by a cost or compares it to a probability threshold; a pure ranker does not need it. And calibration measured on the validation set drifts with the base rate in production, for every column alike.
Neither behaviour is "correct". A tree that predicts last year's maximum for a record-breaking day is wrong; a linear model that predicts a negative price is wrong differently. The honest answer is a monitor on inputs outside the training range and a fallback for them, whichever family serves.
Irrelevant is not the danger — leaky is. Every column will seize a feature that carries the answer, and the more capable the model the more efficiently it does so; robustness to noise says nothing about robustness to leakage, which only a point-in-time audit provides.
Every column is wrong somewhere, and the question that finds where is the same for all of them: how much data, of what modality, at what latency, with what interpretability requirement, judged by which metric, at what cost to train and serve. A model family chosen before those are answered is a guess with a library name.