5 lessons

AI & Agent Performance

The newest latency budget: time to first token, tool-call chains, step counts, retries and context size — where agent latency, cost and quality trade against each other.

SymptomSignalMeasurementHypothesisEvidenceRoot CauseChangeValidationRegression Check

Every lesson below starts from an observable symptom and ends with the measurement that proves the fix worked. Numbers carry a label saying whether they were measured, estimated, simulated or invented to show a shape.

Where an Agent Run Actually Spends Its Time

An agent run is a chain of network round trips nobody wrote explicitly: model call, tool call, model call, tool call. The wall clock is dominated by that chain, so step count is the variable that matters most and the one least likely to appear on a dashboard.

Symptom · Users watching a spinner for a task that "should be quick", with wide run-to-run variance: the same question takes 8 seconds once and 45 the next time.
Inside One Model Call: Queue, First Token, Generation

A model call is three different waits with three different causes. Provider scheduling you cannot control, time to first token that scales with your prompt, and generation that scales with your output — and only two of those are yours to shorten.

Symptom · Model calls that take seconds with wide variance, and no way to tell whether the provider is busy, the prompt is too large, or the response is too long.
Reading an Agent Run as a Trace
▶ lab

Model 2.2s, search 0.8s, model 1.6s, database tool 0.2s, model 1.4s — 6.2 seconds in a straight line. The question a waterfall answers is which of those steps are sequential because the data requires it, and which are sequential because that is the order the model happened to emit them.

Symptom · An agent run whose duration is the sum of its parts, with every step waiting for the previous one even when several of them are independent lookups.
What One Agent Run Costs, and Which Term Dominates

Cost per run is tokens times price times steps — and the tokens term grows every step, because each tool result is appended to a context that every subsequent call must pay for again. That quadratic-ish growth is why long runs cost far more than their step count suggests.

Symptom · A model spend line that grows faster than usage, with no single expensive feature to point at — and per-run costs that vary by an order of magnitude for superficially similar requests.
The Agent Returned 200 OK and the Answer Was Wrong

Rate, errors and duration all look healthy while the agent confidently tells a customer something untrue. Traditional service metrics measure whether the machinery ran; agent systems need metrics for whether the task was actually accomplished.

Symptom · Healthy dashboards, no alerts, and a steady trickle of support tickets about answers that were wrong, incomplete, or confidently invented.