Dynamic Context Assembly
Build the context fresh at every step from state, retrieval and tool results; keep the stable prefix byte-identical for caching; and prefer code over templates once logic appears.
One context per step, not one per session
A chat transcript that grows forever is the default and the worst option. In an agent, each step has a different job — pick a tool, interpret a result, decide whether to stop — and the ideal context for each job differs. Assemble the context per step from three inputs: the orchestrator state (goal, plan, step index, scratchpad), fresh retrieval for the current sub-question, and the results of the tools just called.
This is what makes long-horizon agents viable. Step 40 does not need steps 3–35 verbatim; it needs the goal, a summary of progress, the current sub-task, and the last result. Per-step assembly keeps every call short and focused, and it turns "the agent forgot the goal" into a bug in state rendering that you can fix deterministically (Workflow State Graph).
Stable prefix, dynamic suffix
Providers cache the key/value computations for a context prefix that exactly matches a previous request, and bill those tokens at a steep discount with lower latency. The catch is *exactly*: a single changed character at position 200 invalidates the cache for everything after it.
So split the context into a stable prefix — system prompt, tool definitions, few-shot examples, long reference material — and a dynamic suffix — memory, retrieval, tool results, state, request. Never interpolate anything that varies (timestamps, user names, request ids) into the prefix. Put the date in the suffix. Keep tool definitions in a fixed order. Across a 30-step run, the prefix is paid for once and reused 29 times.
- Prefix: byte-identical across calls in a session, and ideally across users.
- Suffix: everything that changes, ordered per Context Ordering & Lost in the Middle.
- Verify with the provider's cache-hit metrics in your traces; a 0% hit rate means something in the prefix is leaking variance.
Templates vs code
Templates (Jinja, f-strings, prompt files) are fine while assembly is substitution: fill in a name, insert a document list. They stop being fine the moment you need conditionals, loops with budgets, sorting by score, truncation with markers, or different layouts per step type. That logic in a template is untestable and invisible in code review.
Write the assembler as ordinary functions: one per section, each pure (inputs → string, token count), composed by a top-level build_context(step, state). Templates can still hold the static prose. Unit-test the functions with fixed inputs, snapshot the assembled output for key scenarios, and diff those snapshots in CI — a prompt change is a code change (Regression Gates and Online Evaluation).
1type Step = { kind: 'choose_tool' | 'interpret' | 'finalize'; index: number }2 3const PREFIX = [SYSTEM_PROMPT, renderTools(TOOLS), FEW_SHOT].join('\n\n') // never changes4 5export function buildContext(step: Step, state: AgentState, lastResult?: string): string {6 const suffix: string[] = [7 `## Goal\n${state.goal}`,8 `## Progress\n${state.progressSummary}`,9 ]10 if (step.kind === 'choose_tool') suffix.push(`## Next sub-task\n${state.plan[step.index]}`)11 if (step.kind === 'interpret' && lastResult) suffix.push(`## Tool result\n${truncate(lastResult, 1500)}`)12 if (step.kind === 'finalize') suffix.push(`## Collected facts\n${state.facts.map((f) => '- ' + f).join('\n')}`)13 suffix.push(`## Now\n${new Date().toISOString().slice(0, 10)} · step ${step.index + 1}/${state.plan.length}`)14 return PREFIX + '\n\n' + suffix.join('\n\n')15}Keeping it honest
Dynamic assembly is powerful enough to hide bugs. Two habits keep it honest. First, log the final assembled string for every call with its section token counts (Trace Inspection: Debugging from a Trace); the assembler is code, and code needs observability. Second, keep assembly deterministic given the same state — no randomness, no wall-clock in the prefix — so that a failing trace can be replayed exactly.
And resist the urge to make the assembler clever. Retrieval that depends on the model's own previous output, adaptive budgets that shift per step, and self-modifying instructions each add a probabilistic feedback loop. Start with fixed budgets and fixed layouts, and let evals justify every dynamic rule you add.
Key points
- Assemble a fresh, focused context per step from state, retrieval and the latest tool results.
- Long runs stay cheap because step 40 sees a summary of progress, not steps 3–35.
- Keep the stable prefix byte-identical so prefix caching pays for it once per session.
- Never interpolate variable data (dates, ids, names) into the prefix.
- Move from templates to code as soon as assembly needs conditionals, loops or budgets.
- Log the final context per call and keep assembly deterministic for replay.
When to use — and when not to
- Multi-step agents and workflows with more than a few model calls per task.
- Sessions with a large, unchanging system prompt or tool set where caching pays.
- When different step types need different views of the same state.
- Single-call features; a template and a budget are enough.
- When state is tiny and history is short — per-step assembly adds machinery without benefit.
- Do not make assembly depend on model output in ways you cannot replay or test.
Failure modes
- A timestamp in the system prompt drops the cache hit rate to zero and doubles cost.
- Progress summary omits a decision from an early step; a later step redoes or contradicts it.
- Template logic grows conditionals nobody can test; a layout bug ships unnoticed.
- Assembler reads state lazily and two steps see inconsistent snapshots.
- Retrieval per step uses the model's paraphrase rather than the user's wording and drifts off-topic.
Tradeoffs
More code than a transcript, but each call is shorter, cheaper and reproducible.