The question this answers
How does a service four hops down learn that it has 300ms left rather than its configured 30 seconds?
Every hop that honours the convention knows an upper bound on how long its result remains useful, and no hop waits longer than its caller. The bound is conservative: propagated time excludes the transit the request has already spent unless clocks are synchronised, so a hop’s belief about its remaining budget is always at least as large as the truth.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A receiving service knows the remaining duration it was told and how long it has been holding the request, measured on its own monotonic clock. It does not know how long the request spent in the network on the way to it, whether the caller has already given up, or whether the number it was given was computed honestly. Its budget is therefore an over-estimate — safe for deciding to stop, unsafe for concluding the caller is still there.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Relative remaining time, not an absolute deadline
There are two ways to express a deadline on the wire, and the choice is a clock question. An absolute deadline (deadline = 2026-08-25T10:00:02.500Z) is precise and composes trivially — every hop compares it with its own clock. It also requires every machine in the chain to agree on what time it is, and they do not: a receiver whose clock is 400ms fast concludes the request has already expired and rejects work that had most of its budget left. The failure is silent, load-independent, and appears as an unexplained error rate on one host.
A relative remaining duration (timeout: 1700ms) needs no clock agreement at all. Each hop starts its own monotonic timer on receipt and subtracts its own elapsed time before passing the remainder down. This is why gRPC puts a relative grpc-timeout on the wire rather than a timestamp, and it is the right default for the same reason.
The cost of relative encoding is that the transit time from caller to callee is invisible: the callee starts counting when it receives, so the budget it believes it has is larger than the real remaining time by one network hop. Over four hops with 5ms links, the leaf over-estimates by roughly 20ms. That over-estimate is in the safe direction — you occasionally do a little work whose result arrives too late, rather than refusing work that would have made it. Subtract a transit allowance at each hop if you want the number to be tight.
Note also that a relative deadline must be re-derived from a monotonic clock, never from wall time. A wall clock that steps backwards during an NTP correction can hand a request more budget than it started with; one that steps forward expires everything in flight. This is Never Measure a Duration With the Wall Clock in its most operationally expensive form.
| Encoding | Needs synced clocks | Counts transit | Fails as |
|---|---|---|---|
| Absolute timestampassumption | Yes — skew directly corrupts the budget | Yes | One host rejects everything, or waits far too long. Rate correlates with a single machine. |
| Relative remaining durationprotocol | No | No — over-estimates by one hop | Slight over-run per hop; safe direction |
| Relative + per-hop transit allowancetypical | No | Approximately | Slightly conservative; occasional early refusal |
| Nothing propagated (local defaults)typical | No | No | Every hop waits its own default; work outlives the caller by tens of seconds |
The chain breaks at the hop that forgets
Propagation is an all-or-nothing property along a path. One service that does not read the incoming deadline restores its own default for itself *and everything below it* — the budget is not merely lost at that hop, it is reset. A chain with nine correct services and one forgetful one has the failure behaviour of a chain with no propagation at all beneath that point.
The places it is most often lost are the boundaries where the request stops being a request. A queue: publishing to a broker and consuming later almost never carries the deadline, and the consumer may pick the message up long after it expired. A thread-pool handoff: the context lives in a thread-local or an async-local, and dispatching to a worker without copying it drops the budget. A language or protocol boundary: an HTTP call from a gRPC service, a Lambda invocation, an FFI call — anywhere the header has to be manually restated. A cache or proxy layer that reconstructs the request rather than forwarding it.
This makes deadline propagation an infrastructure concern rather than an application one. It belongs in the client library, the server interceptor and the framework middleware, so that the default behaviour of any new service is correct and the effort is in opting out. Systems that ask each team to remember it have propagation in roughly the services that were written by people who had read this lesson.
The signature of a dropped deadline is worth memorising because it is unmistakable: a latency histogram with a spike at exactly the default timeout value. A dense band at 30.0s means a hop is waiting its default, not its budget.
1// Monotonic, not wall clock: an NTP correction must never change a budget.2const now = () => Number(process.hrtime.bigint() / 1_000_000n)3 4interface Ctx { deadlineAtMono: number }5 6export function contextFromHeader(headerMs: number | undefined, fallbackMs: number): Ctx {7 // No header means an unbudgeted caller. Use a bounded fallback, never Infinity.8 return { deadlineAtMono: now() + (headerMs ?? fallbackMs) }9}10 11export function remainingMs(ctx: Ctx): number {12 return ctx.deadlineAtMono - now()13}14 15export function outgoingHeader(ctx: Ctx, transitAllowanceMs = 10): number {16 const left = remainingMs(ctx) - transitAllowanceMs17 if (left <= 0) throw new DeadlineExceeded('no budget left')18 return left19}20 21// Every entry point checks first. A request that arrives already expired is22// refused in microseconds — the cheapest load shedding a system can do.23export function guard(ctx: Ctx, ownP50Ms: number): void {24 const left = remainingMs(ctx)25 if (left <= 0) throw new DeadlineExceeded('expired on arrival')26 if (left < ownP50Ms) throw new DeadlineExceeded('insufficient budget; failing fast')27}What propagation makes possible at the leaf
The reason to build this plumbing is not tidiness. A propagated budget converts several guesses into decisions, and all of them happen at the bottom of the stack where the capacity actually is.
A database that receives a 300ms budget can set a statement timeout, refuse a plan it estimates at 4 seconds, and choose a cheaper plan when the expensive one cannot finish in time. A worker dequeuing a job can drop it in a comparison rather than executing it for a requester who left. A cache can decide that a recompute cannot finish inside the budget and serve stale instead — the decision at the heart of One Key Expires and Five Hundred Instances Miss at the Same Millisecond. A service can pick a degraded, cheaper code path when its budget is thin rather than failing at full cost.
There is a second-order effect worth naming: because expired work is dropped, load falls fastest exactly when the system is most overloaded. As queues grow, more requests expire before being dequeued, so more are dropped for free, which drains the queue. Deadline propagation gives a system a natural negative feedback against overload, and it is the reason a budgeted system tends to degrade rather than collapse.
The honest limit: propagation tells a service how long its answer will be *useful*, not whether anyone is still waiting. Knowing the caller has gone requires The Caller Is Gone — Stopping Is Usually Right and Sometimes Unsafe, which is a separate mechanism with its own delivery problems.
WITHOUT WITH edge timeout 2.0s 2.0s db statement_timeout 30.0s (default) 0.6s (from budget) db active queries 4,100 210 db queries with no recipient ~78% ~2% worker jobs executed after expiry 61% 0% (dropped at dequeue) latency histogram spike at 30.0s no spike; capped at budget behaviour under 3x load collapse degrades, recovers when load falls
Key points
- Propagate a *remaining duration* rather than an absolute timestamp, so no clock synchronisation is required.
- Measure remaining time on a monotonic clock; a wall-clock correction can otherwise extend or expire every in-flight request.
- Propagation is all-or-nothing along a path: one hop that ignores the deadline resets it for everything below.
- It is lost at boundaries where the request is reconstructed — queues, thread-pool handoffs, protocol changes — so it belongs in framework code, not application code.
- A propagated budget lets leaves refuse infeasible work, which gives the system negative feedback against overload instead of collapse.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The entry point converts the client’s patience into a remaining duration and stores a monotonic deadline in the request context.
- • Every outgoing call serialises the current remainder, minus a transit allowance, into a header the callee understands.
- • Every server interceptor reads the header, starts its own monotonic timer, and rejects immediately if the value is non-positive.
- • Asynchronous handoffs — queues, worker pools, background tasks — carry the deadline explicitly rather than relying on ambient context.
- • Leaf components use the remainder to size their own timeouts and to refuse work they cannot finish in time.
- • A hop omits the header, restoring its own default for the whole subtree below it.
- • The deadline is computed from wall time, and an NTP step corrupts every in-flight budget at once.
- • An absolute deadline meets a skewed clock, and one host rejects or over-waits on everything.
- • The deadline is stored in a thread-local that does not follow an async continuation, so it is silently absent after the first
await. - • A message sits in a queue for minutes and is processed with the deadline that was correct when it was published.
- • Latency histogram with a dense band at exactly the default timeout — 30.0s or 60.0s — which is the unambiguous signature of a hop waiting its default instead of its budget.
- • One host rejecting a high share of requests with
DEADLINE_EXCEEDEDin under a millisecond while its peers are fine. Absolute deadlines plus a skewed clock; check NTP offset on that host. - • Queue consumers executing jobs long after their requesters left: queue age is high and result-discard rate is high, with completion counts far above what any caller received.
- • Deadline resurrection after a fleet-wide clock correction: in-flight requests either all expire at once (clock jumped forward) or all gain time (jumped back). Correlated across every service simultaneously, which distinguishes it from a load event.
- • Relative deadlines need no clock coordination at all — the point of the design, and why it is the standard choice.
- • Absolute deadlines trade a clock-synchronisation requirement for tighter accounting; only worth it where you already run tightly bounded clocks and can state the bound.
- • The coordination that is genuinely required is organisational: an agreed header name and semantics that every framework in the estate honours by default.
- • A hop that receives an expired budget fails immediately with a distinct, attributable error rather than timing out later.
- • The chain’s bound holds even when individual services are degraded, because the bound travels with the request.
- • Work already in flight downstream continues unless cancellation is also sent — propagation limits *waiting*, not execution.
- • Detect: look for the default-timeout spike in latency histograms, and count requests arriving with no deadline header at all.
- • Contain: enforce a bounded fallback for headerless requests, and make the server interceptor reject expired arrivals before any handler runs.
- • Recover: no recovery action is needed at runtime; budgets are per request and the next request carries a fresh one.
- • Reconcile: work completed after expiry produced results nobody consumed — audit for side effects those executions caused.
- • Verify: trace a slow request end to end and read the remaining budget at every hop. Any hop where it rises is a hop that dropped it.
- • Share of incoming requests carrying a deadline header, per service — the direct measure of propagation coverage.
- • Remaining budget on arrival, as a distribution per service, which shows which hops are being starved by those above.
- • Count of requests refused on arrival for insufficient budget, distinct from requests that timed out while executing.
- • Latency histograms inspected for bands at exact default values — the cheapest way to find a hop that ignores the budget.
- • Any chain of three or more hops, especially crossing team or language boundaries where defaults differ.
- • Systems with expensive leaves — databases, search indices, model inference — where refusing infeasible work recovers meaningful capacity.
- • Anywhere overload is a recurring failure mode, because dropping expired work is the cheapest shedding available.
- • Fire-and-forget asynchronous work whose value does not expire, where a deadline forces artificial failures on work that could have completed later.
- • Chains where one hop’s latency legitimately dominates and varies hugely; a propagated budget will refuse work that would have succeeded.
- • Where the plumbing cost is real and the chain is two hops deep — a single edge timeout may be sufficient and much simpler.
- • Static per-hop timeouts summing to the client budget: no propagation needed, brittle when the call graph changes.
- • Absolute deadlines with tightly synchronised clocks, if you already operate bounded-error clocks and want transit accounted for exactly.
- • Server-side statement or query timeouts configured per endpoint: coarse, but it stops the worst leaf-level waste with no plumbing at all.
- • Asynchronous job semantics with progress and cancellation instead of a deadline, for work that is genuinely long-running.
Deadline propagation: a budget nobody can see is not a budget
What people believe, and what is true
We set timeouts everywhere, so deadlines are handled.
Independent timeouts bound each hop’s waiting against its own configuration. Only a propagated budget bounds the chain, and only it lets a leaf know its work is already worthless.
Send an absolute deadline — it is more precise.
It is precise only if every clock agrees. With skew, a receiver rejects work that had budget left, and the symptom is confined to one host with no obvious cause.
The framework handles context propagation automatically.
It propagates within one process and one protocol. Queues, thread-pool handoffs, protocol changes and manual client construction each drop it, and the drop is silent.
A propagated deadline means downstream work stops when the caller gives up.
It means downstream knows when its result stops being useful. Stopping in-flight work requires cancellation, which is a different mechanism.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Send the time remaining along with the request, so each service knows how long its answer will still be worth having. Without it, every service waits its own default long after the caller has gone.
Practical
Put a relative remaining-duration header on every call, compute it from a monotonic clock, subtract a small transit allowance per hop, and reject on arrival when it is non-positive. Implement it in the framework so new services get it by default, and carry it explicitly across queues and thread-pool handoffs. Then hunt latency histograms for spikes at default timeout values.
Advanced
The relative encoding is a deliberate trade of accuracy for the removal of a synchronisation assumption — it is the same reasoning as preferring logical clocks to physical ones in Happens-Before: The Only Ordering You Actually Have. The residual error is one network hop per link, always in the direction of over-estimating the budget, which makes the design fail towards doing slightly too much work rather than refusing work that would have succeeded. Whenever you must pick which way a distributed approximation errs, prefer the direction whose failure is wasted effort over the direction whose failure is a wrong refusal.
Apply it
- 💬 Why does gRPC send a relative timeout rather than an absolute deadline?
- 💬 Your p99 histogram has a dense band at exactly 30.0 seconds. What does that tell you?
- 💬 Where does a deadline typically get lost in a service that is otherwise correct?