Where an Agent Run Actually Spends Its Time
An agent run is a chain of network round trips nobody wrote explicitly: model call, tool call, model call, tool call. The wall clock is dominated by that chain, so step count is the variable that matters most and the one least likely to appear on a dashboard.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The wall clock is round trips you did not write
A conventional service has a call graph you can read in the code. An agent decides its own call graph at runtime: the model emits a tool call, the tool runs, the result goes back, the model generates again, and this repeats until it stops. Nothing in your source says "this will take six round trips" — the number is an emergent property of the prompt, the tools available and the specific question asked (The Agent Loop).
This makes agent latency behave unlike service latency in one important way: the variance is enormous and structural rather than incidental. A conventional endpoint with p50 200ms and p99 2s has a tail caused by contention or slow dependencies. An agent with p50 8s and p99 45s usually has a tail caused by *doing more work* — more steps, longer generations, a retry after a tool error. The distribution is multi-modal by step count, and summarising it with a single percentile hides the mechanism (Percentiles: Which One, and How Many Users Is That?).
So the first breakdown to build is not "which component is slow" but "how many steps did this run take, and what did each cost". Once you have that, the optimization targets become visible and they are usually not the ones people reach for first: reducing step count beats making each model call faster, because each step carries a full network round trip plus a generation whose length you only partly control.
What to measure per run
The unit of measurement for an agent is the run, not the request. A run has an outcome (did the task succeed?), a shape (how many steps, of which types) and a cost (tokens in and out, per step). Instrumenting at the HTTP layer gives you none of these — it gives you one long request with a duration, which is the least informative view available (The Agent Returned 200 OK and the Answer Was Wrong).
Six numbers per run cover most diagnosis: total latency, step count, model time, tool time, tokens in/out, and retry count. Splitting model time from tool time immediately answers the most common question, which is whether the agent is slow because the model is slow or because a tool is. In practice it is very often a single tool — a search that takes four seconds, a database query nobody indexed for this access pattern — and that is an ordinary performance problem wearing an AI costume (The Slow Query Workflow).
Record the *distribution* of step count, not the average. A histogram with a long right tail tells you some runs are looping or thrashing, and those runs are simultaneously the slowest, the most expensive and the most likely to have failed at the task. Step-count outliers are the single highest-yield thing to look at in an agent system, and they only exist as a signal if you record step count per run.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Run latency | 38.4s | The symptom. Tells you nothing about which part to attack. | suspect |
| Step count | 11 (p50 for this task type: 4) | Nearly three times the typical shape — this run did extra work, it was not simply slow. | smoking gun |
| Model time (total) | 19.2s across 6 calls | About 3.2s per call. Ordinary; the count is the problem, not the per-call cost. | suspect |
| Tool time (total) | 17.1s across 5 calls | One search tool accounts for 14.8s of it. An ordinary slow dependency (Where the Request Actually Went). | smoking gun |
| Retries | 2 (tool schema validation failed) | Each retry costs a full model round trip; the tool contract is the fix (Tool Schemas). | smoking gun |
| Tokens in / out | 48,200 / 1,940 | Input dominates and grows every step — the context is accumulating (What One Agent Run Costs, and Which Term Dominates). | suspect |
Step count is the multiplier
Because each step is a full round trip plus a generation, latency scales roughly linearly with step count while everything else stays constant. Halving step count is usually a larger win than any per-call optimization available to you, and it is often achievable: give the model a tool that does in one call what it currently does in three, return richer results so a follow-up lookup is unnecessary, or route simple requests to a path with no agent loop at all (Router Architecture).
The second-order effect is worse than linear, because context grows with each step. Every tool result is appended, so step six sends a much larger prompt than step one — which raises time-to-first-token and cost for every subsequent step (Inside One Model Call: Queue, First Token, Generation, Token Budgets). A run that takes twice as many steps therefore costs rather more than twice as much, and the growth compounds in exactly the runs that are already the slowest.
Parallelism is the other lever, and it is under-used. Steps that have no data dependency on each other can be issued concurrently — two independent lookups do not need to be sequential just because the model emitted them in order (Parallel vs Sequential Tool Calls, Reading an Agent Run as a Trace). Whether your framework does this automatically is worth checking; many emit tool calls in parallel only if the model requests them in a single message.
| Steps | Model calls | Approx wall clock | Input tokens (cumulative) | What causes this shape |
|---|---|---|---|---|
| 2 | 2 | ~5s | ~6k | Direct answer with one tool call; the target shape for common questions |
| 4 | 3 | ~11s | ~14k | Typical multi-step: search, then fetch, then answer |
| 8 | 5 | ~26s | ~38k | Exploration — the model is searching rather than knowing where to look |
| 15+ | 9+ | ~60s+ | ~90k+ | Thrashing or looping; frequently ends in a failed or wrong answer (The Agent Returned 200 OK and the Answer Was Wrong) |
Key points
- An agent decides its own call graph at runtime, so step count is an emergent property of the prompt and question — not something the source code states.
- Agent latency is multi-modal by step count; a single percentile over mixed-shape runs describes none of them.
- Measure per run: total latency, step count, model time, tool time, tokens in/out, retries — the model/tool split answers the first question immediately.
- Reducing step count usually beats making each call faster, because each step is a round trip plus a generation plus a larger context for everything after it.
- Step-count outliers are simultaneously the slowest, most expensive and most likely-to-be-wrong runs — they are the highest-yield thing to inspect.
Progressive depth
Overview
An agent run is a loop of model calls and tool calls. Its duration is mostly the number of round trips, not the speed of any single one.
Practical
Instrument per run: latency, step count, model time, tool time, tokens in/out, retries. Split model from tool time first — a single slow tool explains most complaints.
Advanced
Latency is super-linear in step count because context accumulates: each appended tool result raises time-to-first-token for every later call (Inside One Model Call: Queue, First Token, Generation). Plot latency against step count and you get clusters, not a cloud — optimize by moving runs between clusters, not by shaving milliseconds within one.
Internals
The loop is a sequential dependency chain only where data actually flows. Independent lookups can be concurrent, and the serialization is frequently an artifact of the framework or of the model emitting one tool call per message rather than a genuine dependency (Parallel vs Sequential Tool Calls). Meanwhile each round trip carries connection setup, provider-side scheduling and queueing that are invisible to your instrumentation unless you record time-to-first-token separately from total generation time — without that split, provider queueing and slow generation are indistinguishable in your data.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Question → agent: an ambiguous or under-specified request gives the model no clear path, so it explores rather than executing.
- 2Agent → steps: exploration produces extra tool calls, each requiring a model round trip to interpret the result and decide the next move.
- 3Steps → context: every result is appended, so prompt size grows monotonically and each subsequent call has a higher time-to-first-token.
- 4Context → latency: per-step cost rises across the run, making late steps more expensive than early ones and pushing total latency past linear in step count.
- 5Latency → user: a spinner for 40 seconds, and — because exploration correlates with uncertainty — an answer that is also more likely to be wrong.
- • "The model is slow" — split model time from tool time first; a single slow tool accounts for the majority of agent latency complaints, and it is an ordinary dependency problem.
- • "Use a faster model" — this reduces per-call time but not step count, and on a run dominated by tool time or context growth it barely moves the total.
- • "Average run latency is 12s, that is acceptable" — the average of a multi-modal distribution describes no actual run; look at the step-count histogram (The Average Was Fine and Users Were Not).
- • "Streaming made it faster" — streaming improves perceived latency for the final generation only; total wall clock is unchanged and intermediate steps are invisible to the user anyway (Inside One Model Call: Queue, First Token, Generation).
- • "The tail is just provider variance" — check step count on the tail runs first. Structural extra work explains far more agent tail latency than provider jitter does.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Step count per run as a histogram, split by task type — the distribution shape, never the mean.
- • Model time and tool time as separate totals per run, plus per-tool duration so a single slow tool is immediately visible.
- • Tokens in and out per step, to see context accumulation across the run ([[agent-cost]]).
- • Retry count and retry cause per run — schema validation failures, tool errors, timeouts each imply a different fix ([[tool-errors-retries-timeouts]]).
- • Task success rate alongside latency, since the slowest runs are often the failed ones and optimizing them without fixing correctness helps nobody.
- • Reduce step count: give the model higher-level tools that complete in one call what currently takes three, and return richer results so follow-up lookups are unnecessary.
- • Route simple requests away from the agent loop entirely — a classifier plus a direct call is dramatically faster for the common case ([[router-architecture]]).
- • Issue independent tool calls in parallel rather than sequentially, and verify your framework actually does this ([[parallel-vs-sequential-tools]]).
- • Fix the slow tool as an ordinary performance problem — profile it, index the query, cache the lookup ([[slow-query-workflow]]).
- • Bound the loop: a maximum step count with a graceful degraded answer stops a thrashing run from consuming a minute and producing nothing ([[budgets-limits-termination]]).
- • The step-count histogram shifts left, with the long right tail shortened — this is the structural improvement, and it is visible before latency confirms it.
- • p50 and p95 run latency both improve; if only p50 improves, you optimized the common path and left the thrashing runs untouched.
- • Task success rate does not fall — a step-count reduction that trades away correctness is a regression wearing a latency improvement ([[eval-metrics]]).
- • Per-run token totals fall roughly in proportion to the step reduction, confirming context growth was also addressed ([[agent-cost]]).
- • Higher-level tools reduce steps and reduce flexibility — the model can no longer compose primitives in ways you did not anticipate.
- • Routing simple requests around the agent adds a classifier that can misroute, and a second code path to maintain ([[architecture-tradeoffs]]).
- • Parallel tool calls increase peak load on the tools and complicate error handling when one of several fails.
- • Step ceilings guarantee bounded latency and produce incomplete answers for genuinely hard requests; the degraded response needs designing.
- • Track step count per task type as a first-class metric with an alert on distribution shift — prompt and tool changes move it silently.
- • Add step count and token totals to the eval suite so a prompt change that adds a step fails a regression gate rather than a budget review (Regression Gates and Online Evaluation).
- • Alert on the rate of runs hitting the step ceiling, which is the clearest signal of thrashing.
- • Re-check tool latency independently on a schedule; a tool that degrades slowly turns into agent latency with no agent change (Regression or Tuesday? Telling a Real Change from Noise).
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEAll latencies, token counts and step-count shapes are invented to show relationships. Actual values depend on the model, provider, region, prompt size, tool implementation and question — and they change month to month.
- RUNTIME-SPECIFICWhether independent tool calls execute in parallel depends on your agent framework and on whether the model emits them in a single message. Verify rather than assume.