RAGevaluationrecall@kmrrfaithfulnessgolden set

RAG Evaluation

Measure retrieval and generation separately: recall@k, precision, MRR for the retriever; faithfulness and answer correctness for the generator — on a golden set you built from real queries.

Interview question
Progress

Two stages, two evals

A RAG system has two probabilistic stages with different failure classes (RAG Overview), so a single "answer accuracy" number is nearly useless for engineering: when it drops you cannot tell whether chunking broke, the embedding model changed, or the prompt regressed. Evaluate retrieval on its own (did the right chunks come back, how high?), then evaluate generation given a fixed context (did the model answer correctly and only from that context?), then measure the end-to-end number as a sanity check.

Separating them also lets you iterate fast. Retrieval evals need no LLM call, run in seconds, and can be executed on every commit. Generation evals are slower and cost money; run them on a smaller set and on a schedule. Evaluating Agents: Testing Probabilistic Systems covers the general discipline; this lesson covers what is specific to RAG.

Where each metric attaches
Golden queryRetriever + rerankerrecall@k · precision@k · MRRContextLLMfaithfulness · correctnessAnswer
UserLLMAgentToolDataDecisionHumanGuardrail

Retrieval metrics

Each golden query has a set of relevant chunk ids (usually 1–3). Run the retriever and look at the ranked list of ids it returns.

Recall@k: fraction of relevant chunks that appear in the top k. Recall@20 tells you whether the first stage is good enough to feed a reranker; recall@5 tells you whether the LLM will see the evidence. Precision@k: fraction of the top k that are relevant — low precision means the context is mostly noise. MRR (mean reciprocal rank): average over queries of 1 / rank of the first relevant chunk; a relevant chunk at rank 1 scores 1, at rank 4 scores 0.25. MRR is sensitive to ordering in a way recall is not and is the natural metric for Reranking. nDCG generalises this to graded relevance when some chunks are "partially relevant".

A tiny example over 4 queries with one relevant chunk each, found at ranks 1, 3, absent, 2 (with k = 5): recall@5 = 3/4 = 0.75; MRR = (1 + 1/3 + 0 + 1/2) / 4 ≈ 0.46. If a reranker moves the rank-3 hit to rank 1, recall@5 is unchanged and MRR rises to ≈ 0.63 — exactly the improvement reranking is supposed to deliver, visible only in the ranking-aware metric.

Retrieval metrics over a golden set of (query, relevant_ids).
1def retrieval_metrics(golden: list[dict], retrieve, k: int = 5) -> dict:
2    recall, precision, rr = [], [], []
3    for item in golden:
4        ranked = [c.id for c in retrieve(item["query"], k=k)]
5        rel = set(item["relevant_ids"])
6        hits = [i for i, cid in enumerate(ranked) if cid in rel]
7        recall.append(len(set(ranked) & rel) / len(rel))
8        precision.append(len(hits) / k)
9        rr.append(1.0 / (hits[0] + 1) if hits else 0.0)
10    n = len(golden)
11    return {f"recall@{k}": sum(recall) / n, f"precision@{k}": sum(precision) / n, "mrr": sum(rr) / n}

Generation metrics

Given a fixed context and a question, judge the answer on three axes. Faithfulness / groundedness: is every claim in the answer supported by the provided context? Measured by decomposing the answer into claims and checking each against the context with an NLI model or an LLM judge (LLM-as-Judge); citation verification (Citations) is a cheap deterministic proxy. Answer correctness: does the answer match the golden answer? Exact match for short factual answers, an LLM judge with a rubric for longer ones. Relevance: does the answer address the question rather than the context's general topic?

Faithfulness and correctness are independent. An answer can be faithful to a wrong context (retrieval failure passed through honestly) or correct but unfaithful (the model knew the answer from training and ignored the context — which will not survive the next corpus change). Report both. Also measure abstention accuracy: on golden queries with no answer in the corpus, how often does the system say so instead of fabricating?

Keep the generation eval independent of retrieval by feeding it the *golden* context (the known-relevant chunks), not whatever the retriever produced today. That isolates prompt and model changes. Then run the end-to-end variant with live retrieval to see the combined effect.

  • Faithfulness: every claim supported by context — judge or NLI per claim.
  • Correctness: matches the reference answer — exact match or rubric judge.
  • Relevance: answers the question asked.
  • Abstention: says NOT_FOUND when the corpus lacks the answer.

Building a retrieval golden set

The golden set is the asset; the metrics are arithmetic. Start with real queries — from logs, support tickets, or the people who will use the system — not questions you invented while looking at the documents, which are unrealistically well-aligned with the text. Aim for 50–100 queries to start; a few hundred is plenty for most corpora. Include the query styles you expect: short keyword lookups, identifiers, long natural-language questions, ambiguous questions, and 10–20% questions the corpus cannot answer.

For each query, record the relevant chunk ids and a reference answer. Labelling is the expensive part: an engineer with the retriever's top-20 in front of them can mark relevant chunks quickly, and an LLM can pre-label for a human to correct. Store chunk ids *and* the source text, because chunk ids change when you re-chunk; re-map by text match after an ingestion change (Golden Datasets).

Version the golden set, review disagreements, and keep adding every production failure you diagnose — a query that failed in the wild is worth ten synthetic ones.

Running evals as engineering practice

Retrieval evals run on every change to chunking, embeddings, filters, or index parameters — they are the unit tests of the retrieval stage. Generation evals run on every prompt or model change. Both produce a table per commit: recall@5, MRR, faithfulness, correctness, abstention accuracy, plus p95 latency and cost per query. A change that raises recall by 3 points and cost by 40% is a decision, not an automatic win.

In production, sample live traffic and run the same judges asynchronously (Regression Gates and Online Evaluation); track citation verification rates and abstention rates as leading indicators. When numbers move, the two-stage split tells you where to look first.

Key points

  • Evaluate retrieval and generation separately; end-to-end accuracy alone cannot localise a regression.
  • Retrieval: recall@k for coverage, precision@k for noise, MRR/nDCG for ordering.
  • Generation: faithfulness to context, correctness vs reference, relevance, and abstention accuracy.
  • Feed generation evals the golden context to isolate prompt/model changes.
  • Build the golden set from real queries, include unanswerable ones, and grow it from production failures.
  • Run retrieval evals on every ingestion change; report cost and latency alongside quality.

When to use — and when not to

Use it when
  • Before choosing chunk size, embedding model, or retriever — every one of those is a measured decision.
  • On every change to any pipeline stage.
  • When production answers degrade and you need to know which stage broke.
Avoid it when
  • Judging a RAG system from a demo of five hand-picked questions.
  • Using only an end-to-end LLM judge and skipping retrieval metrics.
  • Golden sets written by reading the documents — they overfit to the text.

Failure modes

  • Only answer accuracy is tracked; a retrieval regression is misdiagnosed as a prompt problem.
  • Golden set has no unanswerable queries, so fabrication is never measured.
  • Chunk ids in the golden set invalidated by re-chunking; nobody re-maps them.
  • Faithfulness judged with a lenient prompt; unsupported claims pass.
  • Evals run once at launch and never again.
  • Metrics improved by adding chunks until the context is huge; cost and latency ignored.
← Citations