Contextcontextsummarizationcompressiontool-results

Context Selection & Compression

Decide what earns a place in the window, then shrink it: summarise history, truncate tool output, and extract facts instead of pasting raw dumps.

Interview question
Progress

Selection: the cheapest compression is omission

Before compressing anything, ask whether it should be there at all. Each candidate piece of context should answer a concrete question: does the model need this to take the *next* step correctly? A retrieved chunk with cosine similarity 0.31, a tool result from eight steps ago, and the full text of a policy document the user did not ask about all fail that test.

Selection is where most of the token savings live, and it is deterministic code: relevance thresholds on retrieval scores, a recency window on tool results, and explicit lists of which state fields the model needs at each step. Make these choices in code, log them, and tune them with evals (Evaluating Agents: Testing Probabilistic Systems) — not by feel.

  • Retrieval: keep the top-k after Reranking, and drop anything below a score threshold rather than always filling k slots.
  • Tool results: include the last one or two in full; older ones only as one-line outcomes ("search returned 12 results, picked #3").
  • State: pass the fields relevant to this step, not the whole state object.

Summarising history

Conversation history grows linearly and is mostly stale. The standard pattern is a rolling summary: keep the last N turns verbatim, and replace everything older with a compact summary that is itself updated incrementally rather than regenerated from scratch.

The summary must preserve the things the model will need later: decisions made, constraints the user stated, entity ids, and open questions. It should discard pleasantries, retracted requests, and intermediate reasoning. Write the summariser prompt with that explicit checklist, and evaluate it on the questions users actually ask after a long conversation.

Incremental summary: old summary + newly evicted turns → new summary. Cost stays roughly constant per turn.
1def update_summary(llm, old_summary: str, evicted_turns: list[str]) -> str:
2    prompt = f"""Update the running summary. Keep: user goals, stated constraints,
3decisions, ids/numbers, unresolved questions. Drop: chit-chat, superseded requests.
4Max 200 words.
5 
6CURRENT SUMMARY:
7{old_summary}
8 
9NEW TURNS:
10{chr(10).join(evicted_turns)}"""
11    return llm.complete(prompt, max_tokens=300)
12 
13def window(history, summary, keep_last=6):
14    if len(history) <= keep_last:
15        return summary, history
16    evicted, recent = history[:-keep_last], history[-keep_last:]
17    return update_summary(llm, summary, evicted), recent

Truncating and shaping tool results

Tool output is the most common cause of context blow-up because it is produced by systems that have no idea a model is reading it. A list_files call returns 3,000 paths; an HTTP fetch returns a whole HTML page; a SQL query returns 10,000 rows. None of that should reach the model unedited.

Shape results at the tool boundary, in code: cap rows, strip HTML to text, keep head and tail of long output with an explicit [... 2,842 lines omitted ...] marker, and always tell the model that truncation happened so it can ask for more rather than assume completeness. Design tools to return the shape the model needs (Tool Schemas), not the shape the underlying API happens to produce.

  • Cap by tokens, not characters or rows — 500 rows of ids and 500 rows of descriptions cost very different amounts.
  • Prefer head + tail over head only; errors and totals are usually at the end.
  • Make the truncation visible: an unmarked cut is a silent lie about the data.

Extract facts, not dumps

The highest-leverage compression is semantic: replace a document with the facts you need from it. A 4,000-token order confirmation email compresses to order_id=A7731, status=shipped, eta=2026-08-27, carrier=DHL — twenty tokens that are also easier for the model to use correctly.

Extraction can be a cheap model call with a JSON schema (Structured Outputs) or, better, deterministic parsing when the source is structured. Keep the original reachable (a reference id the model can expand via a tool) so that compression never destroys the ability to verify. This is the same idea as Citations: compact in context, complete on demand.

The trade: every summarisation or extraction step is another probabilistic transformation that can drift (Memory Pitfalls). Compress aggressively, but keep evals that check the compressed form still answers the questions the raw form did.

Key points

  • Omission is the cheapest compression; select in code with explicit thresholds and windows.
  • Rolling summaries keep the last N turns verbatim and update a compact summary incrementally.
  • Shape and truncate tool results at the tool boundary, by tokens, with visible markers.
  • Extracted facts beat raw dumps: smaller, and easier for the model to use correctly.
  • Keep a reference to the original so compression never removes the ability to verify.
  • Every compression step is probabilistic; eval that it preserves what later steps need.

When to use — and when not to

Use it when
  • Conversations longer than a handful of turns, or any multi-step agent run.
  • Tools that can return large or unbounded output (files, HTTP, SQL, search).
  • Pipelines where a downstream step needs a few facts from a large upstream document.
Avoid it when
  • Short, single-turn calls that already fit comfortably — do not add a summariser for its own sake.
  • When exact wording matters (legal clauses, code to be edited): pass the verbatim span, not a summary.
  • When you cannot eval the summary; unmeasured compression is silent data loss.

Failure modes

  • Summary loses a constraint the user stated in turn 2; the agent violates it in turn 30.
  • Silent truncation: the model reasons about "all 50 results" when it saw the first 20.
  • Compressing the user request itself, turning a precise ask into a vague one.
  • Summariser hallucinates a decision that was never made and it persists forever.
  • Extraction schema too narrow, dropping the one field the next step needed.
← Context EngineeringContext Ordering & Lost in the Middle →