ProductionAdvanced

Works in demos, fails in production

“An agent works well in demos but fails unpredictably in production. How would you investigate it?”

What this tests

  • Systematic investigation using traces rather than guesswork
  • Awareness of distribution shift between demo inputs and production traffic
  • Knowledge of production-only failure sources: latency, timeouts, concurrency, stale data, injection
  • Building the eval set from production failures

Answers by level

Read the beginner answer first and notice what is missing.

First make failures observable: per-run traces with every LLM call, tool call, arguments, results, latencies, and token counts, tagged with a run id. Without that, "unpredictable" just means "unobserved". See Tracing Agents and Trace Inspection: Debugging from a Trace. Then sample failing runs and classify them: wrong tool or arguments, tool errors and timeouts, retrieval misses, context overflow, iteration cap hit, injected content, or a correct process with a wrong final judgement.

Demos use curated inputs; production has long messages, typos, multiple intents, unusual document formats, and adversarial content. I compare the input distribution and build a golden set from real failures. Production also introduces infrastructure differences: rate limits and 429s, slower tools causing timeouts, concurrency, stale caches, and different data (an empty result set the demo never hit). See Failure Scenarios: Detection and Runbooks.

Each failure class gets a targeted fix and an eval that would have caught it; I do not touch the prompt until I know which class dominates.

Green flags · Red flags

Green flags
  • Starts with tracing and failure classification
  • Compares demo and production input distributions
  • Names production-only sources: rate limits, timeouts, concurrency, stale data, injection
  • Re-runs failing inputs to measure non-determinism
  • Builds golden set from production failures and adds regression evals
  • Considers reducing agency to a bounded component
Red flags
  • Jumps to prompt tweaks or a bigger model
  • No traces or failure taxonomy
  • No awareness of distribution shift
  • No plan to prevent recurrence

Follow-up questions

F1
You have no tracing in production. What is the first step?
F2
Failures cluster at 9–10 am. What do you suspect?
F3
A failing input passes on re-run. What does that tell you?

Practical scenario

A contract-review agent passed a 30-document demo flawlessly. In production it succeeds on about 70% of documents, and the same document sometimes passes and sometimes fails. There is basic logging but no tracing. Lay out your investigation over the first week, the instrumentation you add, the failure classes you expect to find, and the changes you would make to the release process.

Related concepts · Learn this topic

Tracing AgentsTrace Inspection: Debugging from a TraceFailure Scenarios: Detection and RunbooksRegression Gates and Online Evaluation
← Least privilege for agent toolsAgent that never terminates →