The question this answers
The client gives me 2 seconds. How much of it may each hop below me spend?
A bounded end-to-end response time equal to the client’s deadline, achieved by ensuring every hop’s timeout plus its retries fits inside the budget remaining at that hop. It bounds *waiting*, not execution: downstream work may continue past the deadline unless cancellation is also propagated.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node knows the timeout it was configured with and how long it has personally been waiting. Unless a budget is passed to it, it does not know when its caller will give up, how many hops preceded it, or how much of the original budget is already spent — so it cannot tell the difference between having 1900ms left and having none.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Independent timeouts do not compose
Each team picks a timeout that looks sensible for their own service. The gateway waits 2s because that is roughly the patience of a browser. The order service waits 2s on inventory because inventory is normally fast. Inventory waits 2s on the database. Every number is defensible in isolation and the system built from them has no bound at all.
Follow one slow request. At t=0 the client asks. At t=2000 the client gives up and shows an error. At t=2000 the gateway also gives up — and possibly retries, because from its point of view something failed. Meanwhile the order service is still waiting, inventory is still waiting, and the database is still executing a query for a request whose originator disconnected two seconds ago. Every one of those hops is holding a connection, a thread and a buffer for work that can no longer be delivered.
The correct model is that the deadline is a budget owned by the client and spent by the chain. Each hop consumes some of it — network transit, its own processing, its downstream call — and passes the remainder down. The invariant is simple and almost never checked: each hop’s timeout must be strictly less than the budget it was given. A downstream timeout that is greater than or equal to its caller’s is unconditionally a bug, because the extra time can only ever be spent on work nobody will read.
Doing the arithmetic, including the retries
A budget must account for everything that consumes it, and the item most often forgotten is the retry. "Two attempts of 400ms with a 100ms backoff" is not a 400ms hop; it is a 900ms hop. If the surrounding budget assumed 400ms, the first retry blows the whole chain’s bound.
Network transit also has to be budgeted, and it is not the median RTT — a budget sized on the median is exceeded by definition half the time it matters. Use a high percentile of the observed RTT for each link, and leave explicit slack rather than pretending the numbers are exact.
Working down a chain from a 2000ms client budget: reserve 100ms for client-to-edge transit and rendering, leaving 1900ms at the gateway. The gateway reserves 50ms of its own processing and 50ms of slack, giving 1800ms for its downstream call. The order service is invoked with 1800ms; it spends 60ms on its own logic and calls inventory with 1700ms. Inventory needs two attempts at the database — so its per-attempt timeout is (1700 − 100ms backoff) / 2 = 800ms, and the database is told 800ms.
The result is a strictly decreasing sequence — 2000, 1900, 1800, 1700, 800 — and the fact that it is strictly decreasing is the property worth asserting in code. It is also the number the database can use to decide whether to start a query at all: an 800ms budget against a plan estimated at 3 seconds is a request that should be refused immediately rather than attempted and abandoned.
| Hop | Budget on entry | Own work | Transit + slack | Passed down |
|---|---|---|---|---|
| Clientassumption | 2000ms | render 40ms | 60ms | 1900ms |
| Gatewayassumption | 1900ms | auth, routing 50ms | 50ms | 1800ms |
| Order serviceassumption | 1800ms | validation 60ms | 40ms | 1700ms |
| Inventory (2 attempts + 100ms backoff)assumption | 1700ms | lookup 30ms | 70ms | 800ms per attempt |
| Databasetypical | 800ms | query | — | refuse if plan cost exceeds budget |
What a budget is for, beyond not waiting too long
The obvious benefit is bounded latency. The larger benefit is that a budget turns time into a resource that can be *checked before it is spent*, which unlocks three things that unbudgeted systems cannot do.
Free load shedding. At dequeue, a request whose remaining budget is negative can be dropped with certainty that no value is lost. Under overload this is the highest-quality shedding signal available, and it requires no priority scheme — see Rejecting Work on Purpose — and Rejecting It Cheaply Enough to Help.
Retry decisions that are actually informed. A retry is worth attempting only if the remaining budget covers a full attempt. Without a budget, clients retry with no idea whether the result can still be delivered, which is a large part of how retry load persists into a period where none of it can help.
Feasibility checks. A component that knows a request has 50ms left and that its own p50 is 200ms can fail immediately rather than trying. Failing fast at the leaf frees capacity for the requests that can still be served, and it converts an eventual timeout into an explicit, attributable error.
None of this works if the budget stops at a hop. A single service that drops the header reverts itself and everything below it to its local default, and the whole chain’s bound is gone — which is why Pass the Remaining Budget Down, Not a Fresh One is the mechanism this lesson depends on.
1interface Budget { remainingMs: number }2 3function childBudget(b: Budget, ownWorkMs: number, transitMs: number): Budget {4 const child = b.remainingMs - ownWorkMs - transitMs5 if (child <= 0) throw new DeadlineExceeded('no budget left; do not call downstream')6 return { remainingMs: child }7}8 9// A hop that retries must fit ALL attempts inside its own budget.10function perAttemptTimeout(b: Budget, attempts: number, backoffMs: number): number {11 const usable = b.remainingMs - backoffMs * (attempts - 1)12 const per = Math.floor(usable / attempts)13 if (per <= 0) throw new DeadlineExceeded('budget cannot fund the retry policy')14 return per15}16 17// The invariant worth asserting in CI, not discovering in an incident:18// every downstream timeout is strictly less than the budget that funded it.19function assertDecreasing(chain: number[]): void {20 for (let i = 1; i < chain.length; i++) {21 if (chain[i] >= chain[i - 1]) {22 throw new Error(`hop ${i} timeout ${chain[i]}ms >= caller budget ${chain[i - 1]}ms`)23 }24 }25}Key points
- A deadline is a budget owned by the client and divided across the chain, not a value each hop sets independently.
- Every hop’s timeout must be strictly less than the budget it received; equal or increasing timeouts are unconditionally a bug.
- Retries and backoff are spent from the same budget — two 400ms attempts with a 100ms backoff is a 900ms hop.
- Budget transit at a high percentile of RTT, not the median, and leave explicit slack.
- A budget makes three things possible that are otherwise guesswork: free shedding of expired work, informed retry decisions, and fail-fast feasibility checks.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The entry point sets the total budget from the client’s real patience — a browser timeout, a mobile spinner, an SLO.
- • Each hop subtracts its own expected processing and a transit allowance, and passes the remainder to its callee.
- • A hop that retries divides its remaining budget across its attempts plus backoff, rather than applying its nominal timeout per attempt.
- • Any hop with a non-positive remaining budget fails immediately instead of calling downstream.
- • The resulting sequence of timeouts is strictly decreasing, and that property is asserted rather than assumed.
- • A hop uses its configured default instead of the budget it was given, resetting the chain.
- • A retry policy is added later without adjusting the budget, and the hop now needs more time than it has.
- • Transit is budgeted at the median RTT, so the budget is exceeded on exactly the slow requests it was meant to bound.
- • The client’s real patience differs from the configured budget — a mobile client that gives up at 5s while the chain is built for 2s wastes usable time.
- • Queue time before the handler is not counted, so a request that waited 1500ms is given the full budget as if it had just arrived.
- • Work outliving its caller: database query duration p99 sits well above the client timeout, and a meaningful share of completed queries have no recipient. Visible as active query time greatly exceeding what the request rate and client timeout can account for.
- • Timeout inversion in production: p99 latency of a service exceeds its caller’s timeout, so the caller times out on every slow request while the callee eventually succeeds. Both dashboards look explicable and the pair is broken.
- • Retry blowout: adding a retry to a middle hop raises end-to-end p99 past the client deadline without changing any single service’s own latency. Nobody edited a timeout, so it looks like a mystery regression.
- • Wasted capacity under load: throughput is at capacity, success rate is low, and the difference is requests being served after their client left. Goodput and throughput diverge sharply.
- • Budgets require agreement on one number per request, carried in the request itself — the cheapest form of coordination in this domain, since no consensus and no shared store is involved.
- • It does require organisational agreement on the convention: every service must read the budget and honour it, and one service that does not undoes the chain.
- • No clock synchronisation is needed if the budget is passed as a *remaining duration* rather than an absolute time — see Pass the Remaining Budget Down, Not a Fresh One.
- • A hop that runs out of budget fails explicitly and immediately, which is a better outcome than an eventual timeout and is attributable to a specific hop.
- • End-to-end latency stays bounded even when a downstream is degraded, because the bound comes from the budget, not from the downstream’s behaviour.
- • Work already started downstream may continue past the deadline unless cancellation is propagated — the budget bounds waiting, not execution.
- • Detect: compare each service’s latency distribution against its caller’s timeout. Any overlap is a chain that is already broken for its slow requests.
- • Contain: enforce strictly decreasing timeouts, and check the property in CI from configuration rather than trusting review.
- • Recover: when a downstream is chronically too slow for its budget, either raise the client budget deliberately or reduce the work — never quietly raise one hop’s timeout, which reintroduces the inversion.
- • Reconcile: audit for requests completed after their deadline; those results were discarded and any side effects they caused are unaccounted for.
- • Verify: inject latency at a leaf and confirm end-to-end latency is capped at the client budget rather than at the leaf’s own timeout.
- • Per-hop latency p99 plotted against that hop’s incoming budget — the single chart that reveals inversions.
- • Requests completing after their deadline, counted at each hop. This should be near zero and usually is not.
- • Remaining budget at the leaf, as a distribution. A leaf that regularly receives less than its own p50 is being set up to fail.
- • Goodput versus throughput — divergence is the aggregate symptom of budget problems.
- • Any call chain deeper than two hops, especially where different teams own different hops and set timeouts independently.
- • Systems under load, where budget-based dropping of expired work recovers a large fraction of capacity.
- • Services with a real user-facing latency objective, since the budget is the only mechanism that makes the objective enforceable rather than aspirational.
- • Long-running or streaming operations, where a single end-to-end budget is the wrong model — those need an async job with progress, not a deadline.
- • Chains where hop latencies are wildly variable, so a static division is either too tight for the slow path or too loose to bound anything.
- • When applied so strictly that requests fail on trivial budget overruns during ordinary variance; leave slack, and treat the budget as a bound rather than a target.
- • A single timeout only at the edge, with none below: simple, bounds client waiting, and lets every downstream continue working indefinitely — which is exactly the waste this lesson is about.
- • Fixed per-hop timeouts sized so their sum fits the client budget: static, requires no propagation, and degrades badly when the chain shape changes.
- • Convert to an asynchronous job with a completion notification, removing the deadline problem entirely for work that is genuinely long. API Design owns that pattern.
- • Budget by cost rather than time (rows scanned, calls made), which is more stable than latency for chains dominated by variable work.
Timeout budgets: divide the deadline, do not repeat it
| Hop | Budget on entry | Own work | Transit + slack | Passed down |
|---|---|---|---|---|
| Client | 2.00 s | 40 ms | 40 ms | 1.92 s |
| Gateway | 1.92 s | 50 ms | 40 ms | 1.83 s |
| Order service | 1.83 s | 60 ms | 40 ms | 1.73 s |
| Inventory | 1.73 s | 30 ms | 40 ms | 1.66 s |
2000 → 1920 → 1830 → 1730 → 1660 ms at the database per attempt: (1660 − 100 × 1) / 2 = 780 ms assert(strictly decreasing) -> holds
What people believe, and what is true
Each service should set a sensible timeout for its own dependencies.
Sensible-in-isolation values compose into an unbounded chain. The only sound source of a timeout is the budget you were given.
A generous downstream timeout is safe — it just waits longer.
It waits longer than anyone will read, holding a thread and a connection for work with no recipient. Under load that is the capacity you needed.
Our timeout is 400ms per attempt, so the hop costs 400ms.
With two attempts and backoff it costs 900ms. Retry policies must be funded from the same budget as the attempts.
The client timeout is 2s, so end-to-end latency is bounded at 2s.
Client waiting is bounded at 2s. Downstream execution is bounded only if cancellation propagates; otherwise work continues and consumes capacity.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
The client’s patience is a budget for the whole chain. Divide it as you go down; do not restart it at every hop, or the bottom of the chain will still be working after the top has given up.
Practical
Set the total at the edge from real client patience, subtract own-work and a high-percentile transit allowance at each hop, fund retries and backoff from the same number, and refuse to call downstream on a non-positive remainder. Assert the strictly-decreasing property in CI, and chart each hop’s p99 against its incoming budget.
Advanced
Treat the budget as a resource with a shadow price. Time spent on a request that will miss its deadline has negative value, since it consumes capacity that could serve a request that would make it. That reframing produces the counter-intuitive but correct policies: drop expired work at dequeue, prefer LIFO under overload, and refuse at the leaf when the remaining budget is below the leaf’s own median. Each spends a little fairness to buy a lot of goodput.
Apply it
- 💬 The client waits 2s and every service waits 2s on its dependency. What is wrong, and what does the database observe?
- 💬 A middle service adds one retry and end-to-end p99 breaks the SLO, though no service got slower. Explain.
- 💬 Why is a downstream timeout larger than its caller’s always a bug?