AI & Agent Performance
The newest latency budget: time to first token, tool-call chains, step counts, retries and context size — where agent latency, cost and quality trade against each other.
Every lesson below starts from an observable symptom and ends with the measurement that proves the fix worked. Numbers carry a label saying whether they were measured, estimated, simulated or invented to show a shape.
An agent run is a chain of network round trips nobody wrote explicitly: model call, tool call, model call, tool call. The wall clock is dominated by that chain, so step count is the variable that matters most and the one least likely to appear on a dashboard.
A model call is three different waits with three different causes. Provider scheduling you cannot control, time to first token that scales with your prompt, and generation that scales with your output — and only two of those are yours to shorten.
Model 2.2s, search 0.8s, model 1.6s, database tool 0.2s, model 1.4s — 6.2 seconds in a straight line. The question a waterfall answers is which of those steps are sequential because the data requires it, and which are sequential because that is the order the model happened to emit them.
Cost per run is tokens times price times steps — and the tokens term grows every step, because each tool result is appended to a context that every subsequent call must pay for again. That quadratic-ish growth is why long runs cost far more than their step count suggests.
Rate, errors and duration all look healthy while the agent confidently tells a customer something untrue. Traditional service metrics measure whether the machinery ran; agent systems need metrics for whether the task was actually accomplished.