Practice: Debugging Challenges
Realistic production symptoms. Choose what to investigate; each area reveals a finding. Some findings are red herrings. Find the root cause, then compare your fix.
RAGIntermediate
The document exists but is never retrieved
A user asks about the Q3 refund policy. The exact policy is in the knowledge base, but the agent answers "I could not find that."
ReliabilityIntermediate
The agent loops forever: Search → Analyze → Search → Analyze
A research agent alternates the same two steps — Search, then Analyze — dozens of times without ever answering.
ToolsIntermediate
The agent calls search_web when it should query the database
Asked "how many open orders does customer 4821 have?", the agent calls `search_web` and returns a vague guess.
ContextIntermediate
The assistant gets worse the longer the conversation runs
Early in a session the agent is sharp; after 30+ turns it forgets instructions given at the start and repeats itself.
RAGIntermediate
The answer cites sources that do not say what it claims
The agent produces confident answers with footnote-style citations like [doc-14].
ToolsIntermediate
A tool works in testing but fails intermittently in production
The `create_shipment` tool succeeds most of the time but throws validation errors on maybe 8% of calls.
SecurityAdvanced
A fetched webpage hijacks the agent
After summarizing a page a user linked, the agent emailed the user's recent invoices to an unknown address.
ReliabilityIntermediate
The bill is 20x the estimate a week after launch
Projected LLM spend was ~$400/month; the first week alone cost $2,100.
EvalsAdvanced
The eval suite passes and fails on the same code
Re-running the identical eval suite on unchanged code gives pass rates that swing between 78% and 92%.
Multi-AgentAdvanced
Adding more worker agents makes the system slower
A supervisor delegates to specialist worker agents. Going from 3 to 8 workers increased end-to-end latency.
MemoryIntermediate
The agent insists on facts the user changed months ago
The agent greets the user as living in Berlin and books EU options, though the user moved to Toronto and updated their profile.
ReliabilityIntermediate
The whole product goes down when one API has an incident
During a model-provider incident, every request returned a 500 and users saw a blank error.