The question this answers
Does each hop get its own fresh budget, or do they all share one that is already partly spent?
A request path four hops deep: gateway → checkout → pricing → tax service → currency service. Each hop is configured with a 2-second timeout. The user-facing target is 1 second.
The deadline instant itself, carried in the request context and read by every hop. It is shared state whose correctness depends on clock agreement across processes.
No hop performs work whose result cannot possibly reach the original caller in time, and the total time the user waits is bounded by the budget set at the entry point rather than by the sum of the hops.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
Durations do not compose; instants do
A timeout says "wait at most 2 seconds *from now*". Every hop starting its own 2-second clock is fine in isolation and catastrophic in a chain: four hops means a worst case of 8 seconds against a 1-second target, and every one of the four configurations looks reasonable in its own file. Worse, the failure is not the sum being large — it is that the *later* hops are still working long after the entry point gave up, doing work whose result cannot possibly be used.
A deadline says "be done by 10:31:04.812". It is computed once, at the entry point, from the budget that actually matters, and then travels with the request. Every hop derives its own wait from deadline - now, so the budget shrinks naturally as it is spent. The fourth service does not start a 2-second call with 40 ms remaining; it looks at the clock, sees 40 ms, and either does something fast or fails immediately.
The second benefit is bigger than the arithmetic: a deadline lets a service refuse work it cannot finish. A tax service that needs 200 ms at p99 and receives a request with 40 ms of budget should return "deadline exceeded" without touching its database. That single check removes an entire class of wasted downstream load during a slowdown, and it is only possible if the remaining budget is a value the service can read (Backpressure is the same instinct applied to rate).
How a deadline travels, and where clocks bite
Inside a process, a deadline is a value in the request context and there is no ambiguity: one clock, one instant, exact arithmetic. Across processes it is a header, and now two clocks are involved. If the caller sends an absolute instant and the callee's clock is 300 ms fast, the callee believes it has 300 ms less budget than it does and refuses work it could have completed.
The standard fix is to send the *remaining duration* rather than the absolute instant, and have each receiver immediately convert it back to a local absolute instant using its own monotonic clock. The duration is clock-agnostic; the local instant is stable against wall-clock adjustments. This is why "deadline propagation" in practice usually means a grpc-timeout-style remaining-milliseconds header rather than a timestamp, and it is a distinction worth knowing before you design the header yourself.
Two more rules. Use a monotonic clock for the local arithmetic — a wall clock can jump backwards on an NTP correction and turn a 50 ms remaining budget into a negative one, or worse, a very large positive one. And reserve a margin for the response: if the deadline is the instant the caller stops waiting, work that finishes exactly then still fails, because the response has not been serialised or transmitted. Give each hop a small subtraction for the return trip, or the last hop will consistently produce results that arrive just too late.
1// Entry point: one budget, converted to a local monotonic instant.2function enterRequest(budgetMs: number): Ctx {3 return { deadlineAt: performance.now() + budgetMs } // monotonic, not Date.now()4}5 6function remaining(ctx: Ctx): number {7 return Math.max(0, ctx.deadlineAt - performance.now())8}9 10// Outbound: send a DURATION, and subtract a margin for the response trip.11const RESPONSE_MARGIN_MS = 2012 13async function callDownstream(ctx: Ctx, url: string, body: unknown) {14 const left = remaining(ctx) - RESPONSE_MARGIN_MS15 if (left <= 0) throw new DeadlineExceeded('no budget left before call')16 17 return fetch(url, {18 method: 'POST',19 body: JSON.stringify(body),20 headers: { 'x-request-timeout-ms': String(Math.floor(left)) },21 signal: AbortSignal.timeout(left), // the wait is bounded too22 })23}24 25// Inbound: convert the duration back to a LOCAL instant. No clock comparison.26function acceptRequest(req: Request, defaultMs: number): Ctx {27 const hdr = Number(req.headers.get('x-request-timeout-ms'))28 const budget = Number.isFinite(hdr) && hdr > 0 ? Math.min(hdr, MAX_MS) : defaultMs29 return { deadlineAt: performance.now() + budget }30}31 32// Admission: refuse work that cannot finish. This is the whole payoff.33function admit(ctx: Ctx, p99Ms: number) {34 if (remaining(ctx) < p99Ms) {35 metrics.shedByDeadline.inc()36 throw new DeadlineExceeded('insufficient budget for this operation')37 }38}Which to use, and how they combine
They are not rivals — a deadline is the policy and a timeout is how a single wait is implemented under it. In practice: set a deadline at the entry point from the user-facing budget; at each hop, compute the wait for the next call as remaining - margin; hand that duration to the client library as a timeout; and hand the same deadline to the cancellation source so the work stops rather than merely being abandoned (Timeouts, Cancellation).
Use a plain timeout when there is no chain and no caller budget: a background job polling a queue, a startup health probe, a CLI. There, "wait at most 5 seconds" is exactly the requirement and inventing a deadline adds nothing. Use a deadline everywhere a request passes through more than one component, which in a service architecture is everywhere.
One caution against over-shrinking. A deadline that shrinks at every hop can arrive at a leaf with 3 ms, which is enough for the leaf to refuse and not enough for it to do anything useful. If a large fraction of requests are being refused at the leaves, the budget at the entry point is too small or the chain is too deep — the deadline is reporting a design problem accurately, and lengthening it only converts refusals back into timeouts.
| Dimension | Timeout (duration) | Deadline (instant) |
|---|---|---|
| What it says | Wait at most N ms from now | Be finished by instant T |
| Composes across hops | No — worst case is the sum | Yes — the budget shrinks as it is spent |
| Downstream knows the real budget | No, it starts fresh | Yes, it can read what is left |
| Enables refusing work up front | No | Yes — the single biggest benefit |
| Clock sensitivity | None | Send a duration, convert locally with a monotonic clock |
| Right for | Single isolated waits, background loops, CLIs | Anything crossing more than one component |
| Typical failure | Total latency is the sum of every hop's worst case | Leaves refuse everything because the entry budget was too small |
| Relationship | The implementation of one wait | The policy the waits are derived from |
Key points
- Durations do not compose across hops; instants do. Four hops with a 2-second timeout each is an 8-second worst case behind a 1-second target.
- A deadline computed once at the entry point and carried with the request lets every hop derive its own wait from what is actually left.
- The payoff is admission control: a service that knows only 40 ms remain can refuse before touching its database.
- Send a remaining *duration* over the wire and convert it to a local instant with a monotonic clock — never compare timestamps across machines.
- Reserve a margin for the response trip, or the last hop consistently produces answers that arrive just too late.
- A deadline is the policy; a timeout is how one wait implements it. Use both, derived from the same source.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • The entry point converts the user-facing latency budget into a monotonic instant and stores it in the request context.
- • Every operation that waits computes
remaining = deadline - nowand uses it as its timeout. - • Before an expensive step, the service compares the remaining budget with its own known cost and refuses if it cannot finish.
- • Outbound calls send the remaining duration, minus a response margin, as a header rather than as an absolute timestamp.
- • The receiver converts that duration into its own local monotonic instant, so no cross-machine clock comparison ever happens.
- • The same deadline drives the cancellation source, so exceeding it stops the work rather than only stopping the wait.
- • Per-hop timeouts: gateway waits 2 s, checkout waits 2 s, pricing waits 2 s, tax starts its own 2 s at t=800 ms and finishes at t=2 s — 1.2 seconds of work after the user left.
- • Propagated deadline: pricing calls tax with 120 ms remaining; tax knows its p99 is 200 ms and refuses in under a millisecond. The tax database is never touched, and the whole chain fails fast with one clear cause.
- • Clock skew with absolute timestamps: the callee's clock is 300 ms ahead; a request with 400 ms of real budget appears to have 100 ms; the callee refuses work it could have done. The bug is in the representation, not in the timing.
- • No response margin: the tax service finishes at exactly the deadline; serialisation and transmission take 15 ms; the caller has already returned. Work completed and was still wasted.
- • Wall-clock jump: an NTP correction moves the clock back 200 ms mid-request; a wall-clock deadline gains 200 ms of budget out of nowhere and the entry point returns late. A monotonic clock is immune.
- • Over-shrunk budget: 90% of leaf requests are refused with "insufficient budget". Nothing is broken — the entry-point budget is too small for the chain depth, and the deadline is reporting it correctly.
- • A propagated deadline guarantees the total time is bounded by the entry-point budget rather than the sum of the hops.
- • It guarantees each hop can know how much time remains, which is the precondition for refusing work up front.
- • It does NOT guarantee the work stops at the deadline — that still requires cancellation wired to the same instant (Cancellation).
- • It does NOT survive a hop that ignores the header. One service that starts its own fresh budget breaks the chain from there down.
- • It does NOT account for the response trip unless you subtract a margin explicitly.
- • It does NOT tell a caller whether a write happened. "Deadline exceeded" carries exactly the same ambiguity as a timeout (Timeouts).
- • Deadline-based refusal shifts load away from the deep, expensive parts of the system towards fast rejections — the intended effect, and it changes where contention appears.
- • A shared clock read on every check is essentially free with a monotonic source, but a wall-clock syscall on a very hot path is not; read once per operation rather than per check.
- • Aggressive refusal at the leaves can produce a retry wave at the entry point if clients retry on deadline-exceeded, which needs backoff like any other refusal (Thundering Herd).
- • When many requests share one deadline instant — a batch fanned out from one parent — they all expire simultaneously, producing a correlated burst of cleanup.
- • Budget explosion: independent per-hop timeouts summing far past the user-facing target.
- • Wasted downstream work: hops still computing for a caller that has already returned.
- • Clock-skew refusals when absolute timestamps are compared across machines.
- • Wall-clock jumps producing negative or wildly large remaining budgets.
- • Missing response margin, so successful work arrives fractionally too late.
- • Chain break: one service ignoring the propagated budget and restoring per-hop behaviour from there down.
- • Leaf starvation when the entry budget is too small for the chain depth, refusing nearly everything.
- • Any request path crossing more than one component — which, once there is a gateway, is every path.
- • During partial degradation, where refusing work that cannot finish is the cheapest capacity recovery available (Backpressure).
- • When retries exist, because a retry should inherit the *remaining* budget rather than starting a fresh one — otherwise retries alone blow the total.
- • For fan-out, where every branch should share one deadline so the slowest cannot extend the whole operation (Fan-Out / Fan-In: One Request Becomes N).
- • For isolated single waits with no caller budget — a background poller does not need a deadline, and adding one is ceremony.
- • When the chain is deep and the budget is tight: the deadline correctly refuses most requests, which looks like a new failure and is actually an old design problem becoming visible.
- • When only some services honour it, giving a false sense of end-to-end bounding while one hop still runs unbounded.
- • When the deadline is derived from an unrealistic target, so the system spends its capacity refusing rather than serving.
- • Remaining budget at each hop, recorded as a span attribute. The histogram immediately shows which hop consumes the budget (Distributed Tracing in Performance).
- • Deadline-exceeded rate split by hop, and specifically the count refused *before* doing work — that number is the value the mechanism is delivering.
- • End-to-end latency against the entry-point budget: if p99 exceeds the budget, some hop is not honouring it.
- • Work completed after the deadline had passed, which should be near zero and quantifies wasted capacity when it is not.
- • Clock offset between services, if absolute instants are used anywhere — and treat any nonzero offset as a reason to switch to durations (Cross-Region Latency Is Physics, Not Configuration in Performance).
- • A request context must exist and be threaded everywhere, including across async boundaries where ambient storage silently drops it.
- • A header contract must be agreed across services, including its units, its maximum, and what a receiver does when it is absent.
- • Each service needs a credible estimate of its own cost to make refusal decisions, which means it needs its own latency data.
- • The margin values are another tuning surface, and getting them wrong produces failures that look like ordinary slowness.
- • Per-hop timeouts, when the chain is one hop deep. Simpler and entirely adequate there (Timeouts).
- • A hard concurrency limit per service, which bounds resource usage without any time arithmetic — weaker, and it does not stop useless work (Bounding Concurrency).
- • Asynchronous processing: accept the request, return a job id, and remove the shared time budget entirely (The Async Job Pattern in API Design).
- • Shorten the chain. Four hops for one user action is often the actual problem, and a deadline mostly makes that fact measurable (Composed APIs: Aggregating Other Services in API Design).
The deadline expired. What happened to the work?
try:
result = await wait_for(call(req), 300ms) # only the *wait* is bounded
except Timeout:
return 504 # call() is still running, on a worker, right nowWhat people believe, and what is true
A timeout and a deadline are the same thing expressed differently.
They are, for one hop. Across a chain a duration restarts at every hop and an instant does not, which is the difference between an 8-second worst case and a 1-second one.
We should send the deadline as a timestamp so it is unambiguous.
A timestamp is only unambiguous if the clocks agree, and they do not. Send the remaining duration and let each receiver convert it locally.
A deadline stops the work.
It bounds the budget. Stopping the work still requires cancellation wired to the same instant; otherwise the deadline is just a better-computed timeout.