Contextcontexttokenscostsliding-window

Token Budgets

Give every context section a token budget, measure real usage with the tokenizer, do the cost arithmetic before launch, and use sliding windows so history has a fixed size.

Interview question
Progress

Budget per section, not per call

A single "max 100k tokens" limit on the whole call is how contexts silently fill with history. The workable approach is a budget per section, enforced by the assembler: the system prompt gets a fixed allowance, history gets N tokens, retrieval gets M, tool results get K, and the request is never cut.

Budgets make trade-offs explicit and testable. If retrieval quality drops after you shrank its slice from 6k to 3k tokens, that is a measurable regression with a known cause. Without budgets, the same regression arrives as "the agent got worse last week" with no lever to pull.

  • Allocate by priority: request > instructions > latest tool result > retrieval > memory > history.
  • Reserve output space: a 128k window with 120k of input leaves the model 8k tokens to answer and reason.
  • Leave headroom (10–20%) for tokenizer variance and unexpected tool output.

Measure with the tokenizer

Characters and words are unreliable proxies. English prose runs about four characters per token, but JSON, code, URLs, and non-Latin scripts tokenize far worse — a UUID can cost 20+ tokens, and a base64 blob is a catastrophe. Count with the provider tokenizer (or a token-counting endpoint) and log the per-section counts on every call.

Those counts are the raw material for both budgeting and observability (Logging, Metrics and Alerts): p50/p95 tokens per section, per route, per release. A p95 that creeps upward is the earliest warning of a context leak.

Fit sections to budgets by priority; low-priority sections are trimmed first, the request never is.
1BUDGET = {"system": 1500, "memory": 600, "history": 2500, "knowledge": 4000, "tools": 2000, "state": 400}
2 
3def fit_to_budget(parts: list[tuple[str, str]], total: int, count=count_tokens) -> str:
4    out, used = [], 0
5    for name, text in parts:
6        if name == "request":                 # never trimmed
7            out.append(text); used += count(text); continue
8        cap = BUDGET.get(name, 0)
9        if count(text) > cap:
10            text = truncate_to_tokens(text, cap, marker="[... truncated ...]")
11        out.append(text); used += count(text)
12    assert used <= total, f"over budget: {used} > {total}"
13    return "\n\n".join(out)

Cost math, worked example

Do the arithmetic before shipping. Suppose a support agent averages 4 model calls per conversation, each with 9,000 input tokens and 400 output tokens, at 3 USD per million input tokens and 15 USD per million output tokens. Per conversation: input 4 × 9,000 = 36,000 tokens → 0.108 USD; output 4 × 400 = 1,600 tokens → 0.024 USD; total about 0.13 USD.

At 50,000 conversations a month that is about 6,600 USD. Now let history grow unbounded so the average input becomes 25,000 tokens: input cost nearly triples to 0.30 USD per conversation and the monthly bill passes 16,000 USD — with no improvement in answer quality, and slower calls. That is the cost-explosion-after-launch story, and a budget per section prevents it.

Prefix caching changes the numbers: cached input tokens are typically billed at a large discount (often ~90% off), so a 1,500-token system prompt that is identical across calls costs almost nothing after the first call. Structure contexts so the stable part is a literal prefix (Dynamic Context Assembly).

  • Cost per conversation = Σ calls × (input tokens × input price + output tokens × output price).
  • Output tokens are usually 3–10× the price of input; reasoning-heavy outputs dominate faster than you expect.
  • Multiply by realistic volume and by p95, not average — the long tail pays the bill.

Sliding windows

The simplest fixed-size structure for history is a sliding window: keep the last N turns (or the last T tokens) and drop the rest, optionally into a rolling summary (Context Selection & Compression). It is the same discipline as a fixed-size sliding window over an array: constant memory, O(1) update per step, and you always know exactly how much the section costs.

Windows by turn count are easy to reason about; windows by token count are what actually protects the budget, because turns vary wildly in size. Implement the token-based one, and evict whole turns (never split a message) so the model never sees a truncated user message mid-sentence.

Token-bounded sliding window over turns: evict oldest whole turns until the window fits.
1from collections import deque
2 
3class TurnWindow:
4    def __init__(self, max_tokens: int):
5        self.max, self.turns, self.used = max_tokens, deque(), 0
6 
7    def push(self, turn: str) -> list[str]:
8        """Add a turn; return evicted turns (to feed the summariser)."""
9        evicted, t = [], count_tokens(turn)
10        self.turns.append(turn); self.used += t
11        while self.used > self.max and len(self.turns) > 1:
12            old = self.turns.popleft()
13            self.used -= count_tokens(old); evicted.append(old)
14        return evicted

Key points

  • Budget each section separately; a single total limit is how history eats the window.
  • Count tokens with the tokenizer; characters mislead badly on JSON, code and ids.
  • Do the cost arithmetic: calls × (input × price + output × price) × volume, at p95.
  • Output tokens cost several times more than input; cached prefix tokens cost far less.
  • Sliding windows give history a constant size; evict whole turns by token count.
  • Watch p95 tokens per section per release — creep is a context leak.

When to use — and when not to

Use it when
  • Any agent that will run at volume or on long conversations.
  • When choosing between models or context sizes — the numbers decide.
  • Before adding a memory or retrieval source: assign it a budget first.
Avoid it when
  • Prototypes exploring feasibility; add budgets before the first real users, not before the first experiment.
  • Do not budget the user request — it is the one section that must survive intact.
  • Do not rely on character counts as a budget mechanism.

Failure modes

  • History grows until every call hits the context limit and the provider truncates silently.
  • A tool returns base64 or a huge JSON blob and one call costs 100× the average.
  • Budgets set on characters; a code-heavy route exceeds the token limit anyway.
  • No output reserve: the model runs out of room mid-answer.
  • Cost estimated on averages; p95 conversations cost 10× and dominate the bill.
← Context Ordering & Lost in the MiddleDynamic Context Assembly →