Cost blew up after launch
“Costs of your agent grew ten times faster than traffic after launch. How do you find the cause and bring it under control?”
What this tests
- Attributing cost per step, per tool, per task type from traces
- Understanding super-linear cost drivers: context growth, retries, loops, judge calls
- Cost controls: budgets, caching, model routing, compression
- Balancing cost against quality with evals
Answers by level
Read the beginner answer first and notice what is missing.
Cost growing faster than traffic means cost per task grew, so I attribute cost from traces: tokens and dollars per run broken down by step, model, tool, and task type, then look at the distribution rather than the mean; a fat tail of long runs often accounts for most of the spend. Common drivers: context accumulating across steps (history and tool results never compressed), more steps per task on real inputs than in the demo, retries counted in tokens, and evaluation or judge calls running on every production request. See Logging, Metrics and Alerts and Token Budgets.
Controls in order of impact: per-run token and dollar budgets that stop runaway runs; compression of history and tool outputs so tokens per step stay flat; prompt caching for stable prefixes; routing simple steps (classification, extraction) to a small model and reserving the strong model for judgement steps; caching tool results and retrieval within and across runs; and removing steps that evals show add nothing. See Fallbacks, Caching and Model Routing and Context Selection & Compression.
Every cost change is validated against the golden set so quality does not silently drop.
Green flags · Red flags
- Attributes cost per step, tool, task type from traces; looks at the tail
- Identifies context growth and loops as super-linear drivers
- Per-run, per-user, per-day budgets and hourly anomaly alerts
- Model routing, prompt caching, tool result caching, compression
- Cost per successful task as the unit metric, validated with evals
- Replaces model steps with deterministic code where possible
- Only swaps to a cheaper model
- No attribution from traces
- Ignores quality impact of cost changes
- Monthly budget alerts as the only control