Timeouts: The Latency Contract Nobody Writes Down
A timeout is a statement about how long a caller will wait before deciding the answer is worthless. Set it too short and you manufacture load; too long and you tie up capacity waiting for work that stopped mattering minutes ago.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Two ways to get it wrong, in opposite directions
A timeout that is too short converts slowness into failure and then multiplies it. If the dependency's p99 is 800 ms and the timeout is 500 ms, roughly a percent of calls fail — and if each failure is retried, the dependency now receives extra load precisely when it is already slow. The caller has manufactured the outage it was trying to protect itself from (Retry Storms: The Load You Generated Yourself).
A timeout that is too long is quieter and often worse. Every request waiting on a 30-second timeout occupies a worker, a connection and memory for 30 seconds. At even modest arrival rates, a slow dependency will consume every worker in the pool, and the service stops serving requests that have nothing to do with that dependency (Concurrency Limits: An Unbounded Server Is a Slower Server). This is how one degraded downstream takes down a service that only used it for an optional feature.
The honest way to choose is from the dependency's observed distribution and the caller's own budget: a timeout somewhere above the dependency's p99 catches genuine hangs without failing normal requests, and it must fit inside whatever time the caller has left (Latency Budgets: Spending 200 Milliseconds on Purpose). If those two constraints conflict — the dependency's p99 exceeds your remaining budget — no timeout value fixes it, and the call does not belong in the synchronous path.
| Setting | What happens | Signal that reveals it | When it is nonetheless right |
|---|---|---|---|
| Below dependency p99 | Normal-but-slow calls fail; retries amplify load | High retry ratio, dependency load rising with caller errors | The answer is genuinely worthless when late, and a fallback exists |
| Just above p99 | Catches hangs, tolerates ordinary variation | Timeout rate tracks genuine dependency incidents only | The usual default |
| Several seconds | Workers occupied; pool exhausted by one slow dependency | In-flight concurrency pinned; unrelated endpoints degrading | Batch or background work with no waiting user |
| Tens of seconds / none | Service fails wholesale when the dependency degrades | Everything slow, worker pool full, dependency slow | Almost never for interactive request paths |
Budgets must shrink down the chain
A single timeout value applied at every hop is a common and expensive mistake. If a user-facing request has one second to complete and calls three services sequentially, each of those cannot also have one second — the arithmetic does not fit. The correct model is a deadline that propagates: the caller passes the absolute time by which it needs an answer, and each hop computes its own remaining budget from it, subtracting what it has already spent.
The waterfall below shows why the distinction matters. With per-hop timeouts of 1 s each, the request can legitimately take 3 s before anything trips, and the user gave up at 1 s. With a propagated deadline, the third hop knows only 180 ms remain and can either fail fast or return a degraded answer — no capacity is spent on work that was already too late to matter.
Deadline propagation also improves the *quality* of failure. A hop that knows it cannot finish in the remaining time can return a partial result, skip an optional enrichment, or serve slightly stale data, instead of running to completion and discovering the caller is gone. That decision is only available if the remaining time is part of the request context.
The work does not stop when the caller stops waiting
The most commonly overlooked consequence: a client timeout ends the *client's* wait, not the *server's* work. Unless cancellation is propagated, the server continues executing, still holding its database connection, still consuming CPU, still eventually writing a response to a socket nobody is reading. Under sustained overload this is how a system ends up doing 100% doomed work — every request in flight belongs to a caller who left (Queueing: Why Systems Get Slow Before They Get Broken).
Fixing it requires cancellation to be a first-class part of the call path: a context or cancellation token threaded through handlers, checked before expensive steps, and honoured by client libraries and database drivers. Many drivers ignore it entirely, which is worth verifying rather than assuming — the query keeps running on the database long after the application has moved on.
There is also a correctness dimension, not just a performance one. A timed-out write may or may not have been applied, which makes the outcome genuinely unknown to the caller. Retrying it safely requires idempotency, which is a contract decision, not a timeout setting (Retries and Timeouts as Contract Guidance, Idempotency Keys: The Mechanism).
- Propagate cancellation, not just deadlines — check it before expensive work and pass it into every downstream call.
- Verify the driver honours it: many database and HTTP clients ignore cancellation and keep the work running.
- Treat a timed-out write as an unknown outcome, never as a failure — it may have committed (Idempotency Keys: The Mechanism).
- Cap retries and add jittered backoff, so a slow dependency is not handed multiplied load.
- Pair timeouts with a circuit breaker, so sustained failure stops generating attempts entirely (Circuit Breaker).
Key points
- A timeout below the dependency's p99 converts normal slowness into failures and, with retries, into extra load on the slow dependency.
- A long timeout lets one degraded dependency occupy every worker and take down unrelated functionality.
- Per-hop timeouts do not compose: propagate an absolute deadline so each hop knows how much time is actually left.
- A client timeout does not stop server work — without cancellation propagation the system keeps paying for abandoned requests.
- A timed-out write is an unknown outcome, so safe retry needs idempotency rather than a shorter timeout.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Dependency → caller: p99 rises from 200 ms to 3 s while the caller's timeout stays at 1 s.
- 2Timeout → retries: a large share of calls now exceed the timeout; each failure triggers up to three retries with minimal backoff.
- 3Retries → dependency: effective request rate against the struggling dependency roughly triples, deepening its own queue (Queueing: Why Systems Get Slow Before They Get Broken).
- 4No cancellation → capacity: timed-out calls continue executing server-side, holding connections and workers for the full duration.
- 5Root cause → team: the timeout and retry policy converted a dependency slowdown into a self-amplifying outage across two services.
- • "The dependency is failing" — most of its errors are timeouts generated by the caller's own policy, and much of its load is retries.
- • "Increase the timeout so calls stop failing" — that turns failures into occupied workers and spreads the outage to unrelated endpoints.
- • "Retries improve reliability" — against a saturated dependency they reduce it, by adding load exactly when capacity is scarce.
- • "The request ended when the client timed out" — without cancellation the server is still working, still holding resources.
- • "A timed-out write failed" — it may well have committed; treating it as failed and retrying without idempotency causes duplicates.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Compare the configured timeout against the dependency's measured p99 and p99.9 — a timeout below p99 is a decision to fail normal requests.
- • Track retry ratio (retried ÷ original) per dependency; a rising ratio during a slowdown is amplification in progress.
- • Measure worker or connection occupancy attributable to each dependency, to see which one can exhaust the pool.
- • Count requests completed after their caller disconnected — the direct measure of wasted work.
- • Record remaining-budget at each hop when deadlines are propagated, to find hops that routinely start work they cannot finish.
- • Set each timeout from the dependency's measured distribution and the caller's remaining budget — above p99, inside the budget.
- • Propagate absolute deadlines through the call chain so each hop computes its own remaining time.
- • Implement and verify cancellation propagation, including into database drivers and HTTP clients.
- • Cap retries, use exponential backoff with jitter, and add a circuit breaker for sustained failure ([[circuit-breaker]]).
- • Move dependencies whose p99 does not fit the budget out of the synchronous path entirely ([[async-job-pattern]]).
- • Force the dependency slow in a controlled test and confirm the caller degrades partially rather than failing wholesale.
- • Verify retry ratio stays near baseline during the induced slowdown — that is the evidence amplification is broken.
- • Confirm worker occupancy for that dependency is bounded and unrelated endpoints keep serving normally.
- • Check that cancelled requests actually stop: no queries continuing on the database after the caller disconnected.
- • A timeout above p99 means genuinely hung calls are detected later, holding resources for longer before release.
- • Deadline propagation requires plumbing through every layer and cooperation from libraries that may not support it.
- • Circuit breakers add a failure mode of their own — an open circuit rejects requests the dependency might have served, and tuning thresholds badly causes flapping.
- • Moving a dependency out of the request path removes the latency problem but introduces asynchronous state and a completion contract.
- • Alert on retry ratio per dependency, which detects amplification before the dependency's own error rate does.
- • Assert timeout-versus-p99 relationships in configuration review, so a dependency that gets slower surfaces a stale timeout.
- • Include an induced-slow-dependency scenario in resilience testing rather than only testing hard failure (Load Test Shapes: The Shape Is the Hypothesis).
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe chain waterfall and the failure numbers are constructed to show how per-hop timeouts fail to compose. They are not measurements.
- WORKLOAD-SPECIFICThe right timeout depends on the dependency's latency distribution, the caller's budget and whether a useful fallback exists. There is no portable default, and copying a value from another service is how most bad timeouts get set.
Misconceptions
Apply it
Where the depth lives
A timeout tells the caller nothing about whether the operation was applied — the request, the work, or the reply may have been lost. This is why safe retry requires idempotency at the contract level, and why "timeout means failure" is one of the most expensive simplifications in distributed programming.