RAGcontextgroundingtoken budgetdeduplicationabstention

Context Construction & Grounding

Turning ranked chunks into a prompt: order, deduplicate, fit the token budget, and instruct the model to answer only from what it was given — or to say it cannot.

Interview question
Progress

From a ranked list to a prompt

The reranker hands over a list of chunks with scores. The context builder decides what the model actually reads: which chunks make the cut, in what order, with what labels, under what instructions. This step is deterministic code, it is cheap, and it is where a surprising share of generation failures are created or prevented.

Think of the prompt as a document you are writing for a reader with no memory and limited attention. You would not hand a colleague 12 overlapping excerpts in random order with no headings and ask them to "answer from this". The model deserves the same care.

Deduplication and ordering

Deduplicate first. Overlapping chunks from the same section, the same paragraph indexed under two document versions, and boilerplate that matched because it appears everywhere all waste tokens and dilute attention. Drop exact and near-duplicates (a Jaccard or embedding-similarity threshold between selected chunks works), and when two chunks are adjacent in the source, merge them into one contiguous passage.

Order deliberately. Models attend more reliably to the beginning and end of a long context than to the middle (Context Ordering & Lost in the Middle). Two common strategies: most relevant first, so the answer is near the instructions; or most relevant at both ends ("sandwich"), with weaker chunks in the middle. Grouping chunks by source document, in source order, also helps the model reconstruct the narrative instead of reading shuffled fragments.

Label every chunk with a stable identifier and its provenance — [3] Refund policy > Annual plans (updated 2025-11-02) — so the model can refer to it and so Citations have something to point at.

  • Drop near-duplicates; merge adjacent chunks from the same document.
  • Put the strongest evidence first (or first and last), never buried in the middle.
  • Group by document, keep source order within a document.
  • Number chunks and include title, heading path, and date.

Fitting the token budget

A context window is not a target to fill. Every extra chunk costs money and latency and *lowers* accuracy once it crosses from evidence into noise. Set an explicit budget for retrieved context — often 1.5–4k tokens for a chat answer — and fill it greedily from the top of the ranked list, stopping when the next chunk would overflow or when its reranker score falls below the threshold you learned from evals.

Reserve space in the budget for the system prompt, conversation history, and the answer itself; Token Budgets covers the accounting. If a single chunk is too long, truncate it at a sentence boundary rather than mid-word, or better, fix the chunker so that never happens.

A deterministic context builder: dedupe, budget, label.
1function buildContext(ranked: Chunk[], budgetTokens: number, minScore: number): string {
2  const chosen: Chunk[] = [];
3  let used = 0;
4  for (const c of ranked) {
5    if (c.score < minScore) break;
6    if (chosen.some((o) => jaccard(o.text, c.text) > 0.8)) continue;   // near-duplicate
7    const cost = countTokens(c.text) + 20;                              // label overhead
8    if (used + cost > budgetTokens) continue;
9    chosen.push(c);
10    used += cost;
11  }
12  return chosen
13    .map((c, i) => `[${i + 1}] ${c.title} > ${c.headingPath} (${c.updatedAt})\n${c.text}`)
14    .join('\n\n');
15}

Grounding instructions and abstention

The generation prompt must state the contract explicitly: answer only from the numbered passages; do not use outside knowledge; if the passages do not contain the answer, say so instead of guessing. Without this, the model blends retrieved text with its training data, which produces plausible answers that are wrong about the one thing the user cared about — the specifics of *your* documents.

Abstention is a feature. A RAG system that says "the documentation I have does not cover monthly-plan refunds" is more useful and more trustworthy than one that invents a policy. Make the abstention path explicit in the prompt and in the output schema (a found: boolean field or a fixed phrase), test it with questions whose answers are deliberately absent from the corpus, and route abstentions to a fallback — a search UI, a human, or a broader retrieval.

Quoting tightens grounding further: ask the model to quote the exact sentence it relied on before answering. Quoting forces the model to locate evidence rather than summarise vibes, and it gives you a string you can verify against the retrieved text (Citations). The cost is a few dozen extra output tokens.

A grounding prompt with an explicit abstention path.
1SYSTEM = """You answer questions using ONLY the numbered passages provided.
2Rules:
3- If the passages do not contain the answer, reply exactly: NOT_FOUND.
4- Do not use prior knowledge, even if you believe you know the answer.
5- Before the answer, quote the sentence(s) you relied on, prefixed by the passage number.
6- Keep the answer under 120 words."""
7 
8def generate(question: str, context: str) -> str:
9    return llm(system=SYSTEM, user=f"{context}\n\nQuestion: {question}")

Conflicts and stale evidence

Retrieved chunks disagree more often than you expect: two policy versions, a draft and a final, a forum answer contradicting the official doc. If the context builder cannot resolve this (prefer the newer version, prefer doc_type = official), instruct the model to surface the conflict rather than pick silently: "Passage [2] (2024) says 14 days; passage [5] (2025) says 30 days." Dates and document types in the labels are what make that possible.

Key points

  • The context builder is deterministic code; treat it as a first-class component with tests.
  • Deduplicate and merge adjacent chunks before anything else.
  • Put the strongest evidence at the start (or start and end); group by document.
  • Set a token budget for retrieved context and stop at the score threshold, not at the window limit.
  • Instruct the model to answer only from the passages and to abstain when they do not answer.
  • Ask for quotes; they make grounding checkable.

When to use — and when not to

Use it when
  • Every RAG pipeline — there is no version without a context builder.
  • Corpora with overlapping chunks or multiple document versions.
  • Domains where a wrong answer is worse than no answer.
  • Any system that must show its evidence.
Avoid it when
  • Filling the context window because it is available — more chunks past the threshold hurt.
  • Letting the model pick "the best" of 30 chunks — that is the reranker's job.
  • Skipping abstention in a customer-facing product.

Failure modes

  • Duplicate chunks consume half the budget and crowd out the decisive passage.
  • Right chunk buried in the middle of 15; model answers from a weaker one at the top.
  • No grounding instruction; model answers from training data that contradicts the docs.
  • No abstention path; every unanswerable question gets a confident fabrication.
  • Two document versions in context, model silently picks the stale one.
  • Chunk labels missing, so the answer cannot be traced to a source.
← RerankingCitations →