RAG Overview
Retrieval-Augmented Generation fetches relevant passages at query time and puts them in the prompt so the model answers from evidence instead of from memory.
What RAG actually is
RAG is a two-stage system. Retrieval finds the handful of passages most likely to contain the answer; generation asks an LLM to answer using only those passages. The model is not trained on your data — it reads a few thousand tokens of it at request time, the way a person reads the relevant page of a manual before replying.
Every RAG system, whatever the vendor calls it, is the same pipeline: documents are parsed, split into chunks, embedded into vectors, written to an index, then at query time a retriever pulls candidates, an optional reranker orders them, a context builder packs the best into the prompt, and the LLM writes the answer.
The key engineering fact: the LLM can only be as correct as the context it receives. If the right passage is not in the prompt, no amount of prompt polishing produces the right answer.
What RAG is for
RAG earns its place when the answer lives in a corpus the model has never seen or that changes faster than you can retrain: internal wikis, product docs, tickets, contracts, a codebase. It gives you freshness (update the index, not the model), provenance (you can show which passage the answer came from), and access control (retrieve only what this user may see).
It is also the cheapest way to make a general model behave like a domain expert without fine-tuning — as long as the domain knowledge is *lookup-shaped*: "what does clause 4.2 say", "what is the retry policy for the payments service".
- Question answering over private or frequently changing documents.
- Any answer that must cite its source — compliance, legal, support.
- Per-tenant knowledge where each customer sees only their own data.
- Reducing hallucination on factual questions by supplying the fact.
What RAG is not for
RAG retrieves *passages*; it does not compute. "What was total revenue across all 400 invoices" is a SQL question, not a retrieval question — top-k retrieval will return 5 invoices and the model will sum those. Similarly, "summarise the whole 300-page report" needs a map-reduce over the full document, not a top-k lookup.
RAG also does not teach the model a *style* or a *skill*. If you want the model to write in your house voice or follow a complex procedure, that is prompting or fine-tuning. And if the answer is already reliably in the model's training data (general programming knowledge, public history), retrieval only adds latency and noise.
Escalation order from Choosing the Right Abstraction applies: plain code → single LLM call → structured output → tool calling → RAG → workflow → agent. Do not reach for RAG when a database query or a well-written prompt would do.
- Aggregations, counts, joins — use a database and give the model a query tool.
- Whole-corpus summaries — iterate over documents, do not retrieve top-k.
- Behaviour change (tone, format, procedure) — prompt or fine-tune.
- Tiny corpora that fit in the context window — just include them.
Two failure classes
When a RAG system answers wrongly, the first diagnostic question is always: was the right passage in the context? This splits every bug into two classes with different fixes.
Retrieval failure: the correct passage was never retrieved, or was ranked below the cutoff. Causes: bad chunking split the fact, embedding model mismatch, keyword-heavy query (an error code, an ID) that dense search cannot match, metadata filter excluded it, or the fact was never ingested. Fix on the retrieval side; changing the prompt will not help.
Generation failure: the passage was in the context and the model still answered wrongly — ignored it, blended it with prior knowledge, misread a table, or was distracted by an irrelevant neighbour. Fix on the generation side: tighter grounding instructions, fewer and cleaner chunks, ordering, a stronger model, citations.
Because the fixes are disjoint, RAG Evaluation measures the two stages separately. A team that only measures final-answer accuracy cannot tell which half is broken and ends up tuning prompts to compensate for a broken retriever.
1def answer(question: str, k: int = 5) -> dict:2 q_vec = embed(question)3 hits = index.search(q_vec, k=k) # [(chunk, score), ...]4 context = "\n\n".join(f"[{i+1}] {c.text}" for i, (c, _) in enumerate(hits))5 prompt = (6 "Answer only from the numbered passages. "7 "If they do not contain the answer, say so.\n\n"8 f"{context}\n\nQuestion: {question}"9 )10 reply = llm(prompt)11 # Log retrieved ids: without this you cannot tell a12 # retrieval failure from a generation failure later.13 log.info("rag", question=question, chunk_ids=[c.id for c, _ in hits])14 return {"answer": reply, "sources": [c.id for c, _ in hits]}Key points
- RAG = retrieve relevant passages at query time + generate an answer constrained to them.
- The LLM cannot be more correct than the context it is given.
- Use RAG for lookup-shaped questions over private, changing, or citable corpora.
- Do not use RAG for aggregation, whole-corpus summaries, or behaviour change.
- Every wrong answer is either a retrieval failure or a generation failure — diagnose which first.
- Log retrieved chunk ids on every request; it is the cheapest observability you will ever add.
RAG pipeline simulator
System: Answer only from the sources. Cite [n]. If the sources do not contain the answer, say so. [1] (refund-policy.md) Duplicate charges are refunded automatically when the amount is below 100 EUR. [2] (faq-us.md) US customers: refunds for duplicate charges require contacting support with the transaction ID TXN-… [3] (error-codes.md) Error E-4821: payment captured twice due to gateway retry. Resolve via billing console. Question: I was charged twice — will I get the money back automatically?
When to use — and when not to
- Answers must come from documents the model was not trained on.
- The corpus changes weekly or faster.
- Users need to see the source passage behind an answer.
- Different users are allowed to see different documents.
- The question is an aggregation or join over structured data — use SQL via a tool.
- The whole corpus fits comfortably in the context window.
- The model already knows the answer reliably from training.
- You want to change how the model writes, not what it knows.
Failure modes
- Right passage never retrieved; team tunes the prompt for weeks with no effect.
- Passage retrieved but model answers from prior knowledge that contradicts it.
- Top-k returns near-duplicate chunks, crowding out the one that mattered.
- Index is stale: documents updated, embeddings not re-run.
- Retrieval quality is never measured, so regressions after a chunking change go unnoticed.
Tradeoffs
Moderate build cost; excellent debuggability if you log retrieved chunks. Most of the reliability comes from retrieval quality, not the model.