Inference Decision Tool
Six questions about the prediction — not about the model — and a recommendation of batch, online, streaming or hybrid, with the reasons, the alternatives and what the recommended mode will cost you to run.
The mode is decided by the decision, not by the API. If the email goes out at 06:00, a score computed at 02:00 is exactly as fresh as one computed at 05:59, and the batch version is cheaper, simpler, and — the part that matters most in this domain — computes its features with the same code as training. Online is the default people reach for and the most expensive to run. Answer the six questions for a system you actually have, then change one answer and watch which way the recommendation moves.
Start from a system you recognise
Each preset fills in the six answers. None of them carries its own recommendation — the answers do.
- 1How stale may a prediction be when it is used?
Staleness is the price of batch. If a prediction from last night is still right, nothing forces you to compute it live.
- 2Can the prediction be computed before anyone asks for it?
Batch needs to know the subjects in advance — every customer, every product. A search query is not known until it is typed.
- 3Does it need features that exist only at request time?
The query, the current cart, the device, the session so far: features that cannot be in a feature store because they did not exist a second ago.
- 4How long will the caller wait, in milliseconds?
The latency budget bounds the feature fetch, the model and the network together. Zero means nobody is waiting on the response.
- 5How many predictions a day?
Volume decides cost per prediction. A large batch job amortises model loading over millions of rows; an online service pays for idle replicas.
- 6Is the prediction triggered by events on a stream?
When the trigger is an event — a sensor reading, a transaction — the natural home is a stream processor, with the state and ordering problems that brings.
- — A prediction may be days old, so it can be computed on a schedule and stored.
- — The subjects are known in advance: the job can score every one of them without waiting to be asked.
- — No request-time features are needed, so nothing about the answer changes between the run and the read.
- — 2M predictions a day is cheapest as one job that loads the model once.
- — Nobody is waiting on the response.
- — Staleness: every prediction is as old as the last run, and a user whose situation changed this morning is scored on last night.
- — A store to hold the predictions, and a cache-miss path for subjects the job did not know about.
- — A run that fails at 03:00 leaves yesterday's scores in place — you need to know when that happens.
- online — the freshness requirement drops to minutes or seconds, or the subjects stop being known in advance
- hybrid — a few request-time features would materially change the decision and the caller can wait for a rerank
How to read this page honestly
What the model is, and what it deliberately refuses to be.
- SIMULATEDThe recommendation is a deterministic function of the six answers — no preset carries its own answer. Change one answer and the mode changes with it, for a reason the page shows. It is a decision procedure, not a measurement of any real system's latency or cost.
- SCALE-SPECIFICThe volume thresholds (100k, 1M a day) are the points where the argument changes shape, not where a bill does. A team of two with one model should weight "how many systems do we run" far above "cost per prediction".
- CONTESTEDWhether hybrid is a mode or an admission of two systems is argued. The strongest case for calling it a mode is that most large recommendation and search systems are one; the strongest case against is that the seam between its halves is where train/serve skew lives, and naming it hides that.