Reliability EngineeringoptimizationIntermediate
The bill is 20x the estimate a week after launch
Symptoms
- Projected LLM spend was ~$400/month; the first week alone cost $2,100.
- Traffic is only slightly above the estimate — request volume does not explain the gap.
- Average tokens per request are far higher than in the demo.
demo avg: 3,200 prompt tokens/request, 1 model call prod avg: 28,500 prompt tokens/request, 4.1 model calls/request top cost driver: full document corpus (all 40 chunks) injected every request, no cap model: using the largest model for every call, including classification
Investigate
Inspect areas in any order (0/5 inspected). When you think you know the root cause, commit to it.
Context size per request
Model selection
Caching
Retry policy
Request volume