Toolsretryable errorsexponential backoffjittertimeoutscircuit breaker

Tool Errors, Retries and Timeouts

Classify tool failures as retryable or not, retry with exponential backoff and jitter under a timeout, and surface the rest to the model as observations it can reason about.

Interview question
Progress

Classify before you retry

Tools fail for structurally different reasons, and the right response depends on the class. A 503 from a flaky upstream will likely succeed in 400 ms. A 404 for a ticket that does not exist will never succeed and retrying just costs money. A ValidationError means the model got it wrong and should be re-prompted, not the tool re-run.

The first job of the dispatcher is therefore to sort every exception into one of three buckets before deciding what to do. Getting this wrong in either direction hurts: retrying non-retryable errors multiplies latency and cost; failing fast on transient errors makes the agent look unreliable when the infrastructure merely hiccupped.

  • Retryable (transient): timeouts, connection resets, 429, 502/503/504, lock contention, rate limits. Retry with backoff; respect Retry-After when present.
  • Not retryable (permanent): 400/401/403/404/409, schema violations, business-rule rejections, "insufficient funds". Return to the model as an observation.
  • Model error: arguments failed validation. Re-prompt with the error (Argument Validation); do not touch the tool.
  • Unknown exception types default to not retryable — surprises should fail loudly and get classified explicitly later.

Backoff, jitter and timeouts

Exponential backoff spaces retries at base * 2^attempt so a struggling service gets breathing room. Jitter randomizes each delay so a thousand agents that failed together do not retry together — the "thundering herd" that turns a blip into an outage. Full jitter (random(0, cap)) is the standard choice.

Every tool call needs a timeout, and there are two: a per-attempt timeout (this HTTP call gets 5 s) and a per-operation deadline (all attempts together get 20 s). Without the outer deadline, three retries with backoff can quietly take a minute, and the user is staring at a spinner. Propagate the deadline into the tool so it can cancel work, not just abandon it.

Budget retries per tool, not per agent step: three attempts is a sensible default, five for genuinely flaky third parties, one for anything with side effects unless the tool is idempotent (Idempotency).

Retry with full jitter, per-attempt timeout and an overall deadline.
1import random, time
2 
3RETRYABLE = (TimeoutError, ConnectionError)  # plus HTTP 429/5xx wrappers
4 
5def call_with_retry(fn, *args, attempts=3, base=0.5, cap=8.0, deadline_s=20.0):
6    start = time.monotonic()
7    for attempt in range(attempts):
8        remaining = deadline_s - (time.monotonic() - start)
9        if remaining <= 0:
10            raise TimeoutError("operation deadline exceeded")
11        try:
12            return fn(*args, timeout=min(5.0, remaining))
13        except RETRYABLE as e:
14            if attempt == attempts - 1:
15                raise
16            sleep = random.uniform(0, min(cap, base * 2 ** attempt))  # full jitter
17            time.sleep(min(sleep, max(0.0, remaining)))
18        # non-retryable exceptions propagate immediately

Surfacing errors to the model

Once retries are exhausted or the error is permanent, the failure becomes an observation. Return a structured result — {"ok": false, "error": "not_found", "message": "ticket T-8812 does not exist"} — rather than raising through the loop. The model can then choose: ask the user for the right id, try a different tool, or explain that the action is not possible.

Keep error payloads short and specific. A 4 KB stack trace teaches the model nothing and pollutes the context; the error class plus one sentence is enough. Never include secrets or internal hostnames in the message — it will end up in the transcript and possibly in the user-facing answer (Secrets and Untrusted Output).

Track the number of consecutive failures the model has seen in a turn. After two or three, stop the loop and escalate rather than letting the agent "try one more thing" indefinitely (Failure Scenarios: Detection and Runbooks).

Circuit breakers

Retries protect a single call. A circuit breaker protects the whole system when a dependency is down. It counts failures per tool over a sliding window; once the rate crosses a threshold (say 50 % of the last 20 calls) the breaker opens and every call fails immediately for a cooldown period — no network, no backoff, no waiting. After the cooldown it lets one probe through (half-open); success closes it, failure re-opens it.

For an agent this matters twice over. It bounds latency when a tool is dead, and it gives the model an honest, instant signal — "search_web is temporarily unavailable" — so it can fall back to another strategy (Fallbacks, Caching and Model Routing) instead of burning its step budget on a tool that cannot answer.

Dispatcher decision path for a failed tool call
yesnonoyesyesnoTool callBreaker open?FailsRecord failure in breakerRetryable?Attempts / deadline left?Backoff + jitterStructured error → model
UserLLMAgentToolDataDecisionHumanGuardrail

Key points

  • Sort failures into retryable, permanent, and model-error before deciding what to do.
  • Retry only transient errors, with exponential backoff and full jitter.
  • Two timeouts: per attempt and an overall deadline; propagate the deadline into the tool.
  • Permanent failures become short structured observations the model can reason about.
  • Circuit breakers fail fast when a dependency is down and give the model an honest signal.
  • Retrying side-effecting tools requires idempotency; otherwise retry once at most.

When to use — and when not to

Use it when
  • Any tool that crosses a network boundary — every remote call can time out.
  • Third-party APIs with rate limits or known flakiness.
  • Agents running many tool calls per task where one blip should not fail the whole run.
Avoid it when
  • Retrying validation errors — the tool is fine, the arguments are wrong; re-prompt instead.
  • Retrying non-idempotent writes without an idempotency key (duplicate emails, double charges).
  • Unlimited retries "until it works" — that is an unbounded loop with extra steps.
  • Hiding permanent errors from the model; it will keep calling the tool with the same bad input.

Failure modes

  • Retrying 404 five times with backoff: 30 s of latency for an answer that was known instantly.
  • No jitter: a rate-limited API is hit by every agent in lockstep every 1, 2, 4 s.
  • Per-attempt timeout but no deadline; user waits 90 s while three retries stack.
  • Stack trace returned as observation leaks an internal URL and API key into the transcript.
  • Missing circuit breaker: a dead search API makes every agent turn take its full timeout.
  • Model treats a transient failure as permanent and tells the user the feature does not exist.

Tradeoffs

Complexity
low → high
Latency
low → high
Cost
low → high
Reliability
poor → strong
Debuggability
hard → easy

Small code, large reliability payoff; latency rating reflects that backoff intentionally trades time for success.

← Argument ValidationIdempotency →