Model Serving Architecture
Client → Backend → Model Service → Artifact → Prediction. Where preprocessing runs, model-in-process against model-as-service, versioned endpoints, warm-up and health checks — and the line where the Backend domain takes over.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A prediction has to reach a user through a running system. Where does the model live, who owns the boxes around it, and what does the system have to do before a single request is safe to serve?
A search team embedded their ranking model directly in the backend process — simple, fast, one deploy. Then the model grew, every backend deploy took minutes to load it, a bad artifact took the whole API down with it, and the second team that wanted the model copied the loading code and shipped a different version.
Load the artifact inside the backend process and call predict in the request handler. No network hop, no extra service, no serialisation; the model is a library.
The backend's deploy now includes loading the model; a restart serves errors for as long as the load takes, and rolling deploys of the API are gated by model warm-up, not by the API change.
- The backend's deploy now includes loading the model; a restart serves errors for as long as the load takes, and rolling deploys of the API are gated by model warm-up, not by the API change.
- A bad artifact — a corrupt file, a version that raises on some input — crashes the backend, and the whole API is down for a model problem (Artifact Integrity).
- The model and the backend cannot be scaled or versioned independently: a model rollback is an API deploy, and a second consumer gets a copy of the loading code that drifts.
- Preprocessing lives in the backend for the API and in a notebook for the evaluation job; they diverge, and the offline evaluation measures a different function than production runs (Train / Serve Skew).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The surrounding system ranks search results; the target of the model is relevance. The lesson is about the architecture that delivers the ranking, not the ranking itself.
- The properties being designed for are latency inside the page budget, independent deployability of model and backend, and a single, versioned answer to "which model served this".
- The backend receives a query, retrieves candidates from an index, and needs a score per candidate to rank them; features are a mix of query, candidate and user signals assembled in the backend.
- The artifact is large enough that loading it takes tens of seconds and holding it takes a significant share of a process's memory.
- Two consumers now need the model — the search API and an offline evaluation job — and each runs different preprocessing code.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- The chain is client → backend → model service → artifact → prediction → decision. The backend owns the request, authentication, feature assembly from its own data and the decision it makes with the score; the model service owns loading the artifact by digest, applying the shipped preprocessing, running the model and returning a score with the digest. The artifact owns the weights, the preprocessing state and the feature contract (What a Model Artifact Contains).
- Model-in-process removes a network hop and a serialisation and couples the model's lifecycle to the host process. Model-as-service adds a hop and decouples deploys, scaling, failure and language — the service can be in the model's runtime while the backend is not. The hop costs a millisecond or so on a local network; whether that matters depends on the budget (Latency Breakdown).
- A versioned endpoint means a request names the model and version it wants, or names a stage the registry resolves; the service can hold two versions at once, which is what canary and rollback need (Canary Rollout, Rollback & Fallback).
- A model process is not ready when it starts. It is ready when the artifact is loaded and verified, the first few predictions have run so lazy initialisation and caches are warm, and the replay sample reproduces. Readiness is that condition; liveness is only that the process is not stuck.
Who owns which box
The backend receives the query, authenticates, retrieves candidates, assembles the features it owns and asks for scores; it turns scores into a ranked page and owns the timeout and the fallback. The model service loads the artifact by digest, verifies it, applies the shipped preprocessing, runs the model and returns scores with the digest. The artifact is the contract both sides honour. Everything to the left of the model service is the Backend domain's — request lifecycle, timeouts, retries, health checks — and this domain deep-links it rather than restating it.
The value of drawing the boundary is that each box can be deployed, scaled and rolled back on its own, and that "which model served this request" has one answer: the digest in the response.
In-process or as a service
The choice is about lifecycles, not about performance. In-process is faster by one local hop and couples the model to the host: its load time, its memory, its crashes and its language. As-service costs the hop and buys independence on every one of those, plus a single implementation for every consumer. The search team's pain was not the hop they saved; it was four couplings they did not notice they had accepted.
The comparison assumes the model is large enough to matter. A small tree model in one handler has no lifecycle worth separating, and wrapping it in a service is ceremony.
The API deploy includes a long artifact load; a bad artifact crashes the API; the model and API version together; the evaluation job copies the loading and preprocessing code.
The backend calls `score(version|stage, vectors)` with a deadline; the service loads and verifies the artifact, serves two versions for canary and rollback, gates readiness on warm-up and replay; the evaluation job calls the same service.
Model and backend deploy, scale, fail and roll back independently; there is one preprocessing implementation and one digest per response; and a model failure is a clean error the backend's fallback handles instead of an API outage.
Ready means something
A model process passes through states the load balancer cannot see: listening, loading, verifying the digest, warming up, and finally reproducing the replay sample. Only the last is ready. A readiness probe that returns success at the first state sends traffic into a replica that will time out every request for the next thirty seconds, on every scale-out, every deploy and every crash recovery.
The readiness condition is the assumption serving makes on every replica start. Write it as a check, keep it separate from liveness, and measure how long it takes — that number is the cost of every scale-out and the argument for or against aggressive autoscaling.
No request reaches a replica before its artifact is verified, warm and reproducing the replay sample, and no replica that has failed that check stays in the pool.
holds when Readiness is a real check, separate from liveness; the deploy waits for readiness before shifting traffic; a failed start is a failed replica, not a live one.
breaks when Readiness is wired to the HTTP server binding; a liveness timeout kills slow starts; an autoscaler counts a replica as capacity when it is scheduled rather than when it is ready.
respond Fix the probe wiring first; then decide whether warm-up time justifies pre-warmed capacity or a slower autoscaler.
1let state: 'loading' | 'ready' | 'failed' = 'loading'2 3async function start() {4 try {5 const { bundle, digest } = await loadProduction('search-ranker') // verifies digest + signature, parse-only6 for (let i = 0; i < 20; i++) bundle.predict(bundle.warmupVector) // lazy init, allocator, caches7 const sample = bundle.replaySample()8 for (const row of sample) {9 if (Math.abs(bundle.predict(row.x) - row.expected) > 1e-6) throw new Error('replay mismatch')10 }11 current = { bundle, digest }12 state = 'ready'13 metrics.readyAfterMs.observe(process.uptime() * 1000)14 } catch (e) {15 state = 'failed'; metrics.startupFailures.inc(); throw e // never report ready on a bad artifact16 }17}18 19app.get('/livez', (_, res) => res.sendStatus(200)) // the process is not stuck20app.get('/readyz', (_, res) => res.sendStatus(state === 'ready' ? 200 : 503)) // it is safe to send trafficTwo probes, two questions. A liveness probe with a timeout shorter than the load will kill the process during a legitimate start, so its timeout has to be set from the measured readyAfterMs, not from a default.
How to build it
Most important first.
- Put the model behind a service with a narrow contract: input is the feature vector in the artifact's order (or the raw record, if preprocessing is in the artifact), output is the score, the model digest and the path taken. Every consumer, including the offline evaluation job, calls this service or the same library it wraps, so preprocessing has one implementation.
- Version explicitly: the endpoint accepts a version or a stage, the response carries the digest, and the service can serve two versions concurrently for canary and instant rollback.
- Gate readiness on a real check — artifact loaded and digest verified, warm-up predictions run, replay sample reproduced — and keep the readiness and liveness probes separate so a slow load does not get the process killed and a stuck process does not stay in the pool (Health Checks in the backend sense).
- Let the backend own timeouts, retries, circuit breaking and the fallback decision; the model service should be simple enough that its failure is a clean error the backend can handle (Serving Fallbacks).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Time from process start to ready, and the share of requests that hit a replica during its warm-up. These decide whether scale-out and rolling deploys are safe or whether every restart is a partial outage.
- Score parity between the model service and the offline evaluation job on a shared replay sample — the number that says there is one implementation of the function.
- Not the model service's own p50. The client's p99 through the backend, decomposed by hop, is what the budget is spent against.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every consumer reaches the model through the same service or library, so there is exactly one implementation of preprocessing and one place the digest is decided.
- A replica receives traffic only after readiness — artifact verified, warm-up run, replay reproduced — and the readiness condition is checked, not assumed from process start.
- The backend's timeout, fallback and retry policy around the model call is defined and its behaviour under model-service failure has been exercised.
- Offline: a test that starts the service with a corrupt artifact and asserts it never reports ready; a replay comparison between the service and the evaluation job on the stored sample.
- Online: warm-up duration and requests-during-warm-up as metrics on every replica start; the digest in every response reconciled against the registry.
- Over time: exercise a rollback and a canary in a game day, measuring the client's p99 and fallback rate through the transition.
What can go wrong
- Readiness returns true as soon as the HTTP server binds; the load balancer sends traffic to a replica still loading the artifact, and the first requests time out on every scale-out.
- The model service holds one version, so a rollback is a redeploy with a load time; during that window the backend falls back for every request.
- The backend "just adds one feature" in its own preprocessing before calling the service; the artifact's feature contract is now violated on one path and honoured on the other.
- A liveness probe with a short timeout kills the process during a legitimate long load, and the replica restarts forever.
- A model service is another deployable with its own SLO, scaling and on-call; for a tiny model in a single consumer, in-process is genuinely simpler and the hop is pure cost.
- Serving two versions concurrently doubles memory per replica, which for a large artifact may mean fewer replicas or larger hosts.
- A strict readiness gate lengthens every scale-out; fast autoscaling and safe autoscaling pull in opposite directions (Inference Batching).
- "Model-as-service is the professional architecture." It is the right architecture when the model and the backend need independent lifecycles, or when several consumers exist. A small tree model used by one handler is best as a library with a versioned artifact.
- "The service is healthy — the health endpoint returns 200." Liveness is not readiness. A process that answers HTTP while the artifact is still loading is live and not ready, and sending it traffic is an outage.
- "The backend can preprocess; it has the data." It can, and then there are two preprocessing implementations. The feature contract belongs to the artifact, and whoever assembles the vector has to honour it — with a test.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe chain and the ownership split — backend owns the request and the decision, model service owns loading and scoring, artifact owns the contract — hold regardless of model family or framework; what varies is whether the service is a separate process at all.
- SCALE-SPECIFICFor one small model and one consumer, in-process serving with a versioned artifact is correct and a service is overhead; the service earns its hop once the artifact is large, several consumers exist, or model and backend deploy on different rhythms.
- FRAMEWORK-SPECIFICDedicated model servers provide versioned endpoints, concurrent versions and batching out of the box, and impose their own input formats and readiness semantics; the design here is at the level of what any of them must do, and their defaults have to be checked against it.
Where the depth lives
This domain teaches the model and hands the rest off by name.