RAG retrieves well but answers wrong
“Your RAG system retrieves relevant documents but still produces incorrect answers. How would you debug it?”
What this tests
- Separating retrieval quality from generation quality
- Systematic debugging with traces and per-stage evals
- Knowledge of context construction failure modes: ordering, truncation, conflicting chunks
- Whether they verify the premise ("relevant" by what measure?)
Answers by level
Read the beginner answer first and notice what is missing.
First I verify the premise: "relevant" by whose judgement? I pull traces for 20 failing cases and check whether the retrieved chunks actually contain the answer, not just the topic. Often they are on-topic but the specific fact is in a neighbouring chunk that was cut off at a bad boundary, or the answer requires two chunks and only one made the top-k. That is a chunking or recall problem disguised as a generation problem. See Ingestion: Parsing & Chunking and RAG Evaluation.
If the answer is genuinely present, I look at Context Construction & Grounding: is the relevant chunk buried in the middle of a long context (Context Ordering & Lost in the Middle)? Are there conflicting chunks from different document versions, with the model picking the stale one? Is the context truncated by the token budget before the good chunk? Is the prompt allowing the model to fall back on its own knowledge when the context is ambiguous?
I split evals per stage: retrieval recall@k against a golden set, then generation faithfulness given the correct chunks. Fixing the wrong stage is the most common waste of time in RAG debugging.
Green flags · Red flags
- Verifies that retrieved chunks contain the answer, not just the topic
- Isolates generation by testing with hand-picked perfect context
- Names context construction issues: ordering, truncation, conflicting versions
- Uses per-stage metrics such as recall@k and faithfulness
- Looks at chunking boundaries and stale duplicates
- Questions whether the query type suits RAG
- Jumps to swapping the model or adding more documents
- Only touches the prompt
- No use of traces or a failing-case sample
- Cannot separate retrieval from generation