Observabilityobservabilitydebuggingtracesreplay

Trace Inspection: Debugging from a Trace

When an agent misbehaves, the trace is the evidence: walk it to the first bad decision, reconstruct the context the model saw at that step, compare tool arguments to the schema, then replay with a fix — a loop, not a guess.

Interview question
Progress

The debugging loop

Agent bugs rarely announce themselves with a stack trace. The symptom is a wrong answer, a loop, a refund to the wrong order, or a 40-second response. The root cause is almost always a specific decision at a specific step — the model chose the wrong tool, built the wrong argument, or reasoned from context that was missing, stale or poisoned. Debugging is the process of locating that step and explaining it from what the model was given.

The loop is: reproduce (find or capture the trace) → locate the first bad decision → inspect the context at that step → form a hypothesis about why the model decided that → change one thing → replay → confirm. It is the same discipline as any debugging, with the trace playing the role of the debugger.

Trace debugging loop
yesnoSymptom reportedPull the traceFind first bad decisionInspect context at that stepCompare tool args vs schemaHypothesisReplay with one changeFixed?Add golden case
UserLLMAgentToolDataDecisionHumanGuardrail

Find the first bad decision

Read the trace forward, not backward. The final answer is downstream of everything; the interesting span is the earliest one where the trajectory diverged from what a competent operator would do. In practice: walk the LLM spans in order and for each ask "given what this call received, was its output reasonable?" The first "no" is your target. Everything after it is usually the model coping with a bad state.

Common first-bad-decision signatures: a tool call whose arguments do not correspond to anything in the input (tool-argument-drift); a search whose query dropped the constraint the user gave; a decision to finish before a required tool was called; a repeat of an identical tool call, which marks the start of a loop (agent-loop-not-terminating); a router choosing a branch whose description overlaps the correct one (wrong-tool-selected).

  • Sort spans by start time; ignore the final answer until you have found the divergence.
  • A step that is correct given bad input is not the bug — go one step earlier.
  • Repeated identical tool calls: the bug is in the step that failed to update state from the first result.

Inspect the context and the arguments

Once you have the step, reconstruct the exact context the model received: system prompt (which version?), retrieved chunks (were the right ones there, and where in the order? see Context Ordering & Lost in the Middle), tool definitions (did the description say what you think it says?), prior turns (was a summarisation step dropping the key fact?), and any injected content from tool results (see Indirect Prompt Injection). A large fraction of "the model is dumb" bugs are "the model never saw it" bugs.

Then compare the tool arguments to the schema and the input. Was the argument type coerced ("42" vs 42)? Was a required field defaulted silently? Did the model invent an enum value the schema does not list, and did validation let it through? Diff the argument the model produced against the one a human would have produced from the same context; the diff is your hypothesis.

Walk a trace to the first LLM step whose tool call fails a predicate; print what it saw.
1type Span = { id: string; kind: 'llm' | 'tool'; name: string; startMs: number
2  input?: unknown; output?: unknown; parentId?: string }
3 
4function firstBadStep(spans: Span[], isBad: (call: Span, next?: Span) => boolean) {
5  const ordered = [...spans].sort((a, b) => a.startMs - b.startMs)
6  for (let i = 0; i < ordered.length; i++) {
7    const s = ordered[i]
8    if (s.kind !== 'llm') continue
9    const toolCall = ordered.slice(i + 1).find((t) => t.kind === 'tool')
10    if (isBad(s, toolCall)) return { step: s, toolCall }
11  }
12  return null
13}
14 
15const bad = firstBadStep(trace.spans, (llm, tool) =>
16  tool?.name === 'issue_refund' && (tool.input as { order_id?: string }).order_id !== expectedOrderId)
17 
18if (bad) {
19  console.log('context seen by the model:', JSON.stringify(bad.step.input, null, 2))
20  console.log('argument produced:', bad.toolCall?.input)
21}

Replay and confirm

Replay means re-running the agent from the recorded input with tools mocked to return their recorded outputs, so the only thing that changes is the one thing you changed: a prompt line, a tool description, chunk ordering, a schema constraint. If the replayed trace now makes the right decision at that step, the hypothesis is confirmed. Because the model is stochastic, replay several times; a fix that works 2 out of 5 replays is not a fix.

Finish the loop by adding the case to the golden dataset with the correct expected trajectory (Golden Datasets). The bug you just fixed is the one most likely to recur after the next prompt edit.

  • Replay with recorded tool outputs; live tools make replay non-reproducible.
  • Change one variable per replay; a fix that changes three things teaches you nothing.
  • Confirm across N replays, then promote the trace to a golden case.

Key points

  • The root cause is a specific decision at a specific step; find it by reading the trace forward.
  • The earliest divergence is the target; later steps are the model coping with bad state.
  • Most "model is dumb" bugs are "model never saw it" bugs — reconstruct the exact context.
  • Diff produced tool arguments against schema and input; the diff is the hypothesis.
  • Replay with recorded tool outputs, changing one thing, several times.
  • Every debugged trace becomes a golden case.

When to use — and when not to

Use it when
  • Any wrong answer, loop, wrong tool call or latency outlier reported from production or evals.
  • Investigating a regression after a prompt or model change.
  • Reviewing a sampled trace flagged by negative feedback.
Avoid it when
  • Do not debug from the final answer alone; without the trace you are guessing.
  • Do not change the prompt on a hunch and re-run live; replay with recorded tools first.
  • Do not stop at "the model hallucinated" — ask what context made that the likely output.

Failure modes

  • Reading the trace backward from the answer and fixing a downstream symptom.
  • Context not captured in spans; the step cannot be explained and the fix is a guess.
  • Replaying against live tools; the search index changed and the bug no longer reproduces.
  • Declaring victory after one successful replay of a stochastic fix.
  • Fix shipped without a golden case; the same bug returns after the next prompt edit.
← Tracing AgentsLogging, Metrics and Alerts →