Why more memory is not better
“Why is giving an agent more memory or more context not automatically better?”
What this tests
- Understanding attention dilution and lost-in-the-middle effects
- Cost and latency scaling with context
- Precision versus recall in what enters the prompt
- Ability to design selection and compression instead of accumulation
Answers by level
Read the beginner answer first and notice what is missing.
Three reasons. Quality: models attend unevenly across long contexts; relevant facts in the middle of a long prompt are used less reliably, and irrelevant material competes with relevant material, so precision of what you include matters more than volume. See Context Ordering & Lost in the Middle. Cost and latency: both scale roughly linearly with input tokens on every step of a loop, so a 50k-token context on a 10-step agent is 500k tokens per run. Error surface: every memory entry is a potential stale or wrong fact, and the model treats provided context as trustworthy.
The goal is the smallest context that contains what the task needs. That means Context Selection & Compression: retrieve by relevance to the current step, summarise older turns, keep tool results compact, and expire memories. Token Budgets per section (system, memory, retrieved, history) keep growth in check.
I verify with evals: the same task set with different context sizes, measuring accuracy, cost, and latency. Frequently a 4k curated context beats a 40k dump on all three.
Green flags · Red flags
- Names attention dilution / lost-in-the-middle
- Quantifies cost scaling across loop steps
- Argues for precision of inclusion over volume
- Proposes budgets per section and compression
- Runs ablations comparing context sizes
- Assumes long-context models solve the problem
- Only mentions cost as the downside
- No selection or compression strategy
- No measurement plan