Inside One Model Call: Queue, First Token, Generation
A model call is three different waits with three different causes. Provider scheduling you cannot control, time to first token that scales with your prompt, and generation that scales with your output — and only two of those are yours to shorten.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Three waits, three causes
Split every model call into three parts. Provider scheduling — the gap between your request arriving and the provider starting work, driven by their load and your rate-limit tier, largely outside your control. Time to first token (TTFT) — processing your prompt before any output appears, which scales with input size. Generation — producing output tokens roughly one after another, scaling with how many you asked for.
The reason to separate them is that they have different fixes. High TTFT means your prompt is too large: trim context, compress history, retrieve less (Context Selection & Compression). Long generation means your output is too large: ask for structured output rather than prose, cap the response length, stop asking the model to restate its reasoning. High provider scheduling time means you are being queued: change tier, change region, change model, add a fallback, or retry elsewhere (Fallbacks, Caching and Model Routing).
Without the split, all three present identically as "the model call was slow", and teams reach for the fix that is culturally available rather than the one that matches. Adding retries to a call that is slow because the prompt is 90k tokens makes everything worse — it multiplies the expensive part (Retry Storms: The Load You Generated Yourself).
Streaming changes perceived latency, not total latency
Streaming delivers tokens as they are produced, so the user sees output after TTFT instead of after the full generation. For a user-facing chat response this is transformative — perceived latency drops from "total" to "TTFT", which can be a five-fold improvement in how fast the product feels while changing the actual wall clock by exactly zero.
The important corollary: streaming does nothing for intermediate agent steps. When the model output is a tool call that your code must parse and act on, you need the whole output before you can proceed, so the user waits for full generation regardless. In a six-step agent run, only the final generation benefits from streaming — the other five are invisible waits (Reading an Agent Run as a Trace).
This is why perceived-latency work and actual-latency work must be tracked separately. Streaming the final answer, showing intermediate progress ("searching…", "reading the order record…"), and rendering partial results all improve the experience substantially. None of them make the run shorter, and reporting a streaming rollout as a latency improvement in a performance review confuses two genuinely different achievements.
1change: enabled streaming on the chat endpoint2report: "reduced response latency from 4.2s to 0.9s"3 4what actually changed:5 time to first token 4.2s → 0.9s (unchanged; now visible)6 total generation 4.2s → 4.2s (unchanged)7 tokens, cost, steps unchanged8 9problem: the capacity model is now wrong. Concurrency10planning still needs the 4.2s occupancy per request11([[littles-law]]), and the "latency improvement" gets12double-counted in next quarter's capacity forecast.1change: enabled streaming on the chat endpoint2report: "perceived latency (TTFT) 4.2s → 0.9s.3 Total call duration unchanged at 4.2s.4 Connection occupancy unchanged — capacity5 model unaffected."6 7separately tracked:8 perceived_latency_p95 (TTFT) → user experience9 call_duration_p95 (total) → capacity + cost10 step_count_p95 → structural work11 12next: reduce total by capping output length and13trimming context — those move both numbers.Both statements describe the same deployment. The second one keeps perceived latency and occupancy as separate metrics, which matters because capacity planning depends on occupancy and a streamed request occupies its connection for exactly as long as it did before.
The two levers you own: prompt size and output size
Input size drives TTFT. This is the lever that agent systems degrade silently, because context accumulates across steps and nobody watches it — by step eight the prompt may be five times what it was at step one, and every call in between paid for it (Token Budgets). Trimming retrieved documents, summarising earlier turns, and dropping tool results that are no longer relevant all reduce TTFT directly and reduce cost at the same time (Context Selection & Compression).
Output size drives generation time roughly proportionally, and it is the lever people forget they control. Asking for JSON matching a schema rather than an explanation produces dramatically shorter output (Structured Outputs). Asking the model not to restate the question, not to summarise what it is about to do, and not to explain its reasoning when the reasoning is not needed removes tokens that cost real seconds. Maximum output length is a hard cap worth setting even when you do not expect to hit it.
One important caution about caching: prompt caching, where a provider reuses the processed prefix of a repeated prompt, can reduce TTFT substantially for stable prefixes — but only if your prefix is genuinely stable. Agent systems that interleave changing content early in the prompt defeat it entirely. Whether it applies, how it is billed and what counts as a cache hit are provider-specific and change over time; treat any specific behavior here as something to verify against current documentation rather than to memorize (Fallbacks, Caching and Model Routing).
| Lever | Provider queue | TTFT | Generation | Cost | Risk |
|---|---|---|---|---|---|
| Trim / compress context | no effect | large reduction | no effect | reduces input cost | Dropping context the model needed; quality regression |
| Cap output length | no effect | no effect | large reduction | reduces output cost | Truncated answers if the cap is too tight |
| Structured output instead of prose | no effect | slight increase (schema) | large reduction | reduces output cost | Schema validation failures cost a retry (Tool Schemas) |
| Smaller / faster model | may change tier | reduction | reduction | usually cheaper | Quality drop; verify with evals (Eval Metrics: What to Measure and How) |
| Stable prompt prefix (cacheable) | no effect | possible large reduction | no effect | provider-dependent | Defeated by any early-changing content; verify behavior |
| Streaming | no effect | no effect | no effect on total | no effect | None — but it is perceived latency only |
| Fallback to another provider/region | can reduce | varies | varies | varies | Quality and behavior differences between models |
Key points
- A model call is three independent waits — provider scheduling, prompt processing (TTFT), and generation — with three different fixes.
- TTFT scales with input size and is what agent context accumulation silently degrades across a run.
- Generation time scales with output size, which you control through structured output, length caps and not asking for narration.
- Streaming converts perceived latency to TTFT and changes total wall clock and connection occupancy by nothing at all.
- Provider queueing is not yours to optimize; it is yours to route around with tiers, regions, models and fallbacks.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Agent → provider: a request carrying an accumulated context of tens of thousands of tokens is submitted.
- 2Provider → queue: the request waits for scheduling capacity, adding a variable delay driven by their load and your tier, not by your prompt.
- 3Provider → prompt processing: the full input is processed before any token is emitted, so TTFT rises roughly with input size.
- 4Provider → generation: output tokens are produced sequentially, so total time rises roughly with output length.
- 5Client → measurement: all three arrive as one duration, so the team optimizes whichever cause they assumed rather than the one that dominates.
- • "The model got slower this week" — check input token size first; a prompt or retrieval change that grew context raises TTFT with no provider change at all.
- • "Streaming reduced our latency" — it reduced perceived latency. Total duration and connection occupancy are unchanged, and capacity models depend on the latter (Little's Law as Working Intuition).
- • "We should retry slow calls" — if slowness comes from a large prompt, retrying re-pays the expensive part and adds load; retries help with queueing, not with size (Retries and Timeouts as Contract Guidance).
- • "A bigger context window means we can send more" — the window is a capability limit, not a performance recommendation; every token still costs TTFT and money.
- • "The p50 is fine" — provider scheduling produces a heavy tail, and agent runs multiply per-call tails across many calls (Fan-Out: Waiting for the Slowest of Seven).
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Time to first token as a separate metric from total call duration — without this split, provider queueing and slow generation are indistinguishable.
- • Input and output token counts per call, recorded on the span so they can be correlated with duration ([[trace-anatomy]]).
- • Provider-reported latency or rate-limit headers where available, to separate their queueing from your network time.
- • Per-step prompt size across a run, to detect context growth as the run proceeds ([[agent-cost]]).
- • TTFT distribution rather than mean — provider scheduling produces a heavy tail that a mean will hide ([[tail-latency]]).
- • Trim and compress context aggressively — this reduces TTFT and cost together, and it is the lever agent systems most commonly leave on the floor.
- • Cap output length and request structured output instead of prose for anything the code will parse rather than a human will read.
- • Set an explicit maximum token limit as a guard even where you do not expect to reach it, so a runaway generation cannot consume a minute.
- • Route around provider queueing with fallbacks across models, regions or providers, chosen by measured TTFT rather than by list price.
- • Add streaming for user-visible output, and report it honestly as perceived-latency work rather than as a latency reduction.
- • TTFT falls in proportion to the input-token reduction; if it does not, the bottleneck was provider queueing rather than prompt size.
- • Generation time falls in proportion to output-token reduction, measured as total duration minus TTFT.
- • Quality metrics from the eval suite hold after trimming context — a TTFT win that costs accuracy is not a win ([[evals-overview]]).
- • The TTFT tail, not just the median, improves — a median-only improvement means the queueing component was untouched.
- • Trimming context risks removing information the model needed, trading latency for accuracy in a way only evals can detect ([[context-ordering-lost-in-the-middle]]).
- • Output caps truncate genuinely long answers; the truncation behavior needs designing rather than discovering.
- • Structured output reduces generation time and introduces schema-validation failures that cost a full retry round trip.
- • Multi-provider fallbacks add operational complexity and behavioral differences that make quality inconsistent across the same feature.
- • Alert on p95 input tokens per call; context growth is gradual and invisible without a metric on it (Token Budgets).
- • Include token counts and TTFT in the eval suite so a prompt change that doubles context fails a gate (Regression Gates and Online Evaluation).
- • Track TTFT per model and per region continuously, so provider degradation is attributable rather than mysterious.
- • Re-verify prompt-caching assumptions periodically — provider behavior and billing for cached prefixes change, and a silently broken cache looks exactly like a slow model.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEEvery millisecond figure in the budget is invented to show proportions. Real TTFT and generation rates vary by model, provider, region, prompt size, load and time of day, and change substantially between model releases.
- ENVIRONMENT-SPECIFICProvider queueing depends on your account tier, region and the provider's current load. Prompt-caching behavior and billing are provider-specific and change — verify against current documentation rather than relying on any description here.