Tracing a Prediction
One request id, from the click in the frontend through the backend, the feature service, the model server and the decision, to the outcome event weeks later. Each hop records something specific, and a hop that drops the id is where the next incident becomes a guess.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What does each hop between a user action and an outcome record, and what becomes undiagnosable at the hop where the correlation id is lost?
A customer complained that they were declined for a purchase they had made a hundred times before. Support escalated it. Three teams spent two days establishing which model version had scored it, and never established what features it had seen, because the feature service logs by session and the model server logs by request.
Every service logs what it does with whatever key is natural to it. When something needs investigating, join on timestamps and user id; the logs are all there, and a good engineer can piece it together.
The customer made three purchases in the same minute. Joining on user id and timestamp yields three feature snapshots, three scores and three decisions, and no way to say which belongs to the declined one.
- The customer made three purchases in the same minute. Joining on user id and timestamp yields three feature snapshots, three scores and three decisions, and no way to say which belongs to the declined one.
- The feature service keys on session id, which the model server never saw. The hop from "what features were fetched" to "what score came out" is broken exactly where the incident lives — the model saw something that made it decline, and that something cannot be retrieved.
- The decision service keys on transaction id, minted after the score. The prediction log has no transaction id and the outcome table has no request id; the outcome join that would compute quality for this model version depends on a mapping table one team maintains by hand.
- Two days of three teams is the cost of one complaint. The next incident will be a population, not a person, and the same joins will have to be done for ten thousand requests.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The same fraud decision — approve, review, decline — and the same 90-day chargeback outcome. The tracing target is the customer's single request, reconstructed hop by hop.
- The flagship at
/ml/predictionwalks this path for a recommendation; the mechanics are identical, and the point of walking it is to see which hop decides what.
- Frontend: a click event with a client-generated request id, a session id and a timestamp. Backend: a request log with the request id, the user id resolved from the session, and the downstream calls it made.
- Feature service: a fetch log keyed — currently — on session id, with the feature snapshot it returned. Model server: a prediction log keyed on request id with model version and score. Decision service: an action log keyed on transaction id.
- Outcome pipeline: a chargeback record keyed on transaction id, written by the payments provider up to 90 days later.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A correlation id is minted once, at the earliest hop — the frontend or the edge — and propagated as a header or a field through every downstream call. Each hop writes its record keyed on it. Because the id is the same everywhere, the records of one request can be joined exactly, regardless of concurrency, retries or timing (Correlation IDs: Turning Lines Into a Story in the backend domain has the propagation mechanics).
- Each hop records what only it knows. The frontend knows the user action and the surface. The backend knows the resolved identity and the downstream calls. The feature service knows the vector and its freshness. The model server knows the version, the score and the latency. The decision service knows the action and the policy. The outcome pipeline knows what happened, and when it learned it.
- When a hop drops the id, everything downstream of it can still be joined to each other, and nothing upstream can be joined to it. The break splits the chain into two islands, and the question that crosses the split — "what did the model see?" crosses the feature→model split; "was this decision right?" crosses the decision→outcome split — becomes a timestamp guess.
Six hops, six records
The request passes through six places, and each one knows something the others do not. The value of the trace is that each hop writes down its own piece, keyed on the same id, so the pieces assemble later without guesswork. The pipeline below lists what each hop records, and how each one typically breaks the chain.
The flagship at /ml/prediction walks the same path with "go one layer deeper" at each stage; the point of doing it once by hand is to notice that the interesting failures — a stale feature, a wrong version, a cache-served score, a lost outcome — each live at exactly one hop.
- 1Frontend
Mints the request id. Records the user action, the surface, the client timestamp and the session.
fails by No id minted; the backend mints one and the click event cannot be joined to the decision the user saw.
- 2Backend
Resolves identity, forwards the id in every downstream call, records the calls it made and their status.
fails by A retry re-mints the id; the feature fetch and the score are on different ids and the join is empty.
- 3Feature service
Records the id, the feature-definition version, the snapshot reference, the freshness of each feature and whether any fell back to a default.
fails by Keys on session id instead; "what did the model see" becomes a timestamp guess.
- 4Model server
Records the id, the model version, the preprocessing version, the score, the latency, and whether the score came from a fallback rung.
fails by Logs the score and nothing else; the version has to be inferred from the deploy history.
- 5Decision service
Records the id, the transaction id it minted, the threshold policy, and the action taken.
fails by Records the transaction id only; the outcome can never be joined back to the request without a hand-kept mapping.
- 6Outcome event
Records the transaction id, the outcome, and the time it was observed — joined to the request id through the decision record.
fails by Arrives keyed on something else entirely; quality per model version becomes unmeasurable.
The two hops that break most often are the feature service, built for a session-keyed cache, and the outcome pipeline, owned by a team with no stake in the model. Both breaks are invisible until an incident asks a question that crosses them.
Where the break makes the incident undiagnosable
The customer's complaint needed one question answered: what did the model see that made it decline? That question crosses the feature→model hop, and that is the hop where the id was lost. Everything downstream — version, score, decision — was retrievable in minutes. Everything upstream — the vector — was three candidates in the same minute and a coin toss.
A break has a signature: the questions that cross it become guesses, and the questions that do not remain answerable. That is why a chain with one break feels mostly fine right up to the incident that needs the crossing.
1export async function score(req: ScoreRequest, ctx: RequestContext): Promise<ScoreResponse> {2 const requestId = ctx.requestId // minted upstream; the server refuses to mint its own3 if (!requestId) throw new Error('score called without request id')4 5 const features = await featureClient.fetch(requestId, req.entityId) // keyed on the same id6 const t0 = performance.now()7 const result = model.predict(features.vector)8 await predictionLog.write({9 requestId,10 modelVersion: model.version, // what only this hop knows11 featureDefVersion: features.defVersion,12 preprocessingVersion: model.preprocessingVersion,13 featureSnapshotRef: features.snapshotRef,14 score: result.score,15 latencyMs: performance.now() - t0,16 fallbackRung: result.fromFallback ?? null,17 })18 return { requestId, score: result.score, modelVersion: model.version }19}Two decisions in the code carry the lesson. The server throws rather than minting an id, because a self-minted id is the retry failure in another form; and it returns the model version to the caller, so the decision service can record it beside the transaction id it is about to mint.
The assumption the trace rests on
The chain is only as intact as its weakest hop, and the weakest hop is usually owned by the team with the least reason to care. The assumption is not technical — propagating a header is easy — but organisational: that every team on the path treats the id as a contract, including the one whose outcome event arrives three months later.
The check is cheap and should run continuously: take a request id, ask for every hop, and count the misses. A chain that is checked hourly cannot break for two days without someone knowing.
The request id minted at the frontend is present, unchanged, on the record every hop writes — including the outcome event, via the transaction id the decision service records beside it.
holds when The id is a required field in every service's request contract; retries reuse it; batch paths carry a batch-plus-entity equivalent; the decision record stores both ids; the outcome pipeline joins through the decision record.
breaks when A service re-mints on retry; a cache serves a decision without writing a record; the feature service keys on a different id; the outcome pipeline is migrated and the mapping is lost.
respond Fix the hop that misses; until it is fixed, say out loud that any incident crossing that hop cannot be diagnosed from logs, so that "retrain" is not offered as the answer to a question nobody can ask.
How to build it
Most important first.
- Mint the id at the frontend and require it on every call; the backend rejects or re-mints and flags a request without one. The id goes into every log line as a structured field, never inside a free-text message (Structured Logging: Fields a Program Can Read in the observability domain).
- Give the outcome pipeline the id. The transaction id is minted after the score, so the decision service must record the request id alongside it, and the outcome join goes outcome → transaction → request in one query, not through a hand-maintained mapping.
- Make the feature service key on the request id and record the snapshot reference, so "what did the model see" is one lookup (Prediction Logging).
- Add the model-specific fields to the trace span, not only to the log: model version, feature-definition version, fallback rung. A trace that stops at "called model server, 40 ms" is a service trace; one that says "fraud-v7, fd-v4, from cache" is a model trace (Trace, Span, Attribute, Status).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The fraction of decisions for which every hop's record can be retrieved by request id. This is the number that says the chain is intact, and it should be one.
- The time to reconstruct one prediction end to end. Two days is a broken chain; a query is a working one.
- Per-hop latency is worth having and is a service measurement; it does not say what the model saw.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- One id is minted at the earliest hop and every downstream hop — including the outcome pipeline via the transaction id — records it, so that a request can be reconstructed exactly without timestamp joins.
- Each hop records the fields only it knows: the feature snapshot reference at the feature service, the model and feature-definition versions at the model server, the policy and fallback rung at the decision service, the observation time at the outcome pipeline.
- Retries, batch paths and fallback rungs preserve the id rather than re-minting or omitting it.
- Offline: for a random sample of request ids from last week, run the reconstruction and assert every hop is present; alert when the rate falls.
- Online: a synthetic request with a known id sent through the production path hourly, and its record checked at every hop within the expected delay.
- Over time: pick one real complaint per month and time the reconstruction; if it takes more than a query, the chain has a break and the drill has found it.
What can go wrong
- A retry at the backend re-mints the request id, so the feature fetch is keyed on the first id and the score on the second; the chain is intact and the join returns nothing.
- A batch scoring path — the same model, run nightly over a table — has no request and no id; its predictions reach the decision service keyed on entity, and its outcomes cannot be attributed to a model version.
- The id is propagated everywhere and logged nowhere useful: it is in the trace, the trace is sampled at one in a hundred, and the declined request was not sampled.
- Propagating an id through every service is a cross-team contract; the outcome pipeline, owned by payments, is the team least likely to see why it should carry a model team's id.
- Logging model-specific fields on every span increases trace volume; sampling to control it is exactly what makes the interesting request unavailable.
- A request id per prediction does not exist for batch scoring, and inventing one — a batch id plus entity id — is a second convention to maintain.
- "We have distributed tracing, so we can trace a prediction." The trace ends when the response is returned. The outcome arrives in 90 days and is not in any trace; the chain has to be joinable by id in the logs, not only in the tracer.
- "We can join on user id and timestamp." Until the user does two things in the same second, or a retry moves the timestamp, or the clocks on two services disagree. Exact joins need an exact key.
- "The model server logs the request id, so the model is traceable." The model server knows the score. The features are in the feature service, keyed on whatever it chose, and that is the hop the incident is in.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALOne id propagated through every hop and recorded with the hop's own fields is the mechanism for any online prediction path; the specific hops vary, and a batch path needs an equivalent key by construction.
- SCALE-SPECIFICA single service that fetches features, scores and decides in one process has the chain trivially intact; the problem appears with the second team and the second service, and grows with each hop owned by someone whose incidents do not involve the model.
- SIMPLIFIEDThe walk at
/ml/predictionand the hop list here compress a real path — retries, caches, a queue between decision and payment — into the hops that record something the model diagnosis needs; the shape is the argument, not a reference architecture.
Where the depth lives
This domain teaches the model and hands the rest off by name.