The question this answers
One replica is slow for reasons that have nothing to do with my request. Can I just ask someone else?
The client-observed latency of a hedged request is the minimum of the original and the hedge, so the tail is bounded by roughly the hedge threshold plus one service time — *provided* the slowness is independent between replicas. It buys nothing when the cause is shared: an overloaded dependency, a hot shard, or an expensive query is slow on every replica.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A hedging client knows how long it has been waiting and what the recent latency distribution looked like. It does not know why this request is slow, whether the first replica is about to answer, or whether the second replica is any healthier. Hedging is a bet made in ignorance — which is fine, because the bet is cheap and bounded, and it is why the threshold must come from a live distribution rather than a constant.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The bet, and why it is cheap
Set the hedge threshold at the p95 of the recent latency distribution. By construction, only 5% of requests are still outstanding when the threshold fires, so at most 5% of requests generate a second call and the added load is bounded by 5%. That is the whole economic argument, and it is the reason hedging is one of the few tail-latency techniques you can enable broadly without a capacity review.
The benefit is disproportionate to that cost. If replica slowness is independent — a GC pause here, a cold page cache there — then the probability of both the original and the hedge being slow is roughly 0.05 × 0.05 = 0.0025. The p99.75 of the hedged distribution lands near the p95 of the unhedged one. You spend 5% more requests to remove most of the tail, which is a trade very few optimisations offer.
The threshold must be derived from a live percentile estimate, not a hard-coded constant. A fixed 200ms threshold set when p95 was 180ms becomes catastrophic when p95 drifts to 400ms: now *every* request hedges, load doubles, latency rises further, and the hedging mechanism has become the incident. A hedge rate that tracks 5% is the invariant to hold, and it should be enforced directly — cap the hedge rate and let the threshold follow.
Google’s *The Tail at Scale* reports that in a system where hedging fired after the 95th percentile, a fan-out of 100 saw its 99.9th percentile latency drop from 1,800ms to 74ms for roughly 2% extra requests. Numbers from any specific system are not transferable, but the shape — order-of-magnitude tail reduction for a few percent of load — is the reason the technique is used at all.
| Metric | No hedging | Hedge at p95 | Notes |
|---|---|---|---|
| Extra requestsprotocol | 0% | ~5% | Bounded by the threshold percentile |
| P(both slow)assumption | 5% | ~0.25% | Only under independence |
| Effective p99assumption | tail of one replica | ≈ p95 + one service time | The tail is truncated, not shortened |
| Under shared-cause slownessassumption | slow | slow, plus 5% more load | Hedging actively hurts here |
Idempotency is not optional, and neither is a budget
A hedge is a duplicate request by construction. You are deliberately sending the same operation twice, in flight simultaneously, with no way to know whether the first one has already taken effect. Hedging is only safe for operations where a duplicate is harmless — reads, idempotent writes carrying a client-chosen key, and lookups. Hedging a POST /charge is a design for double-charging customers at a rate of about 5% of traffic.
This is stricter than the requirement for retries, and the reason is worth being precise about. A retry follows a failed or timed-out attempt, so at least there is *some* evidence the first attempt may not have landed. A hedge is sent while the first attempt is still healthy and in flight; the most likely outcome is that both execute. Retries need idempotence to be safe; hedges need it to be *correct at all*.
The second requirement is a budget. Hedging is a positive feedback loop under load: latency rises, more requests cross the threshold, more hedges are sent, load rises, latency rises further. Left unchecked this converges on every request being hedged — a doubling of load during the exact incident that triggered it. The mitigation is the same instrument as in Cap Retries as a Fraction of Traffic, Not as a Count per Request: cap hedge traffic as a fraction of total traffic (commonly 5–10%), and when the budget is exhausted, stop hedging and let the tail be long. A hedge refused by budget is a working system, not a degraded one.
Third, hedge somewhere else. A hedge sent to the same replica that is currently slow is pure waste. This requires the client to know its replica set and to exclude the endpoint it is already waiting on — trivial to state, routinely missed, and it silently converts the whole mechanism into a load generator.
1async function hedgedCall<T>(req: Req, replicas: Endpoint[], budget: RetryBudget): Promise<T> {2 const thresholdMs = liveP95() // tracked, not a constant3 const first = replicas[pickIndex(req)]4 const ctl = new AbortController()5 6 const original = send<T>(first, req, ctl.signal)7 const timer = sleep(thresholdMs)8 9 const winner = await Promise.race([original, timer.then(() => 'HEDGE' as const)])10 if (winner !== 'HEDGE') { ctl.abort(); return winner }11 12 // Only hedge if: the operation tolerates duplicates, the budget allows it,13 // and there is a DIFFERENT replica to ask.14 const other = replicas.find((r) => r !== first)15 if (!req.idempotent || !other || !budget.tryWithdraw()) return original16 17 const hedge = send<T>(other, req, ctl.signal)18 const result = await Promise.race([original, hedge])19 ctl.abort() // cancel the loser: see [[request-cancellation]]20 return result21}Tied requests: hedging without the duplicate work
A refinement removes most of the redundant execution. Instead of waiting for p95 and then sending a second request, send both immediately, each carrying the identity of the other. Whichever server dequeues the request first sends a "you can drop it" message to its twin before beginning execution.
The result is that both servers have the request enqueued — so whichever is less busy starts sooner — while only one usually executes it. The duplicate work is limited to the window between one server starting and the cancellation reaching the other, which on a datacentre network is a fraction of a millisecond. *The Tail at Scale* reports this window costing on the order of 1% extra disk reads while substantially reducing median and tail latency.
The trade is precision for coupling. Tied requests need servers that participate — they must expose cancellation and honour it before starting work — whereas plain hedging is entirely client-side and works against any replica set. Use tied requests inside a system you control end to end; use plain hedging when calling something you do not own.
Both variants depend on cancellation actually working, which is the honest link to The Caller Is Gone — Stopping Is Usually Right and Sometimes Unsafe: a cancel that is lost means both replicas execute, and the safety argument reduces to idempotence again. Cancellation improves the economics; idempotence is what makes it safe.
Key points
- Hedging at the p95 bounds the extra load at 5% by construction, because only 5% of requests are still outstanding at that point.
- Under independent replica slowness, hedging truncates the tail: P(both slow) is the square of P(one slow).
- A hedge is a deliberate duplicate sent while the first attempt is healthy — idempotence is a correctness requirement, not a precaution.
- The threshold must track a live percentile and be capped by a budget, or hedging doubles load during the incident that triggered it.
- Tied requests enqueue on two replicas and cancel on start, reducing duplicate execution to a sub-millisecond window at the cost of server participation.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The client tracks a live latency percentile for the dependency and uses it as the hedge threshold.
- • A request is dispatched to one replica and a timer is armed at the threshold.
- • If the timer fires first, the client checks that the operation is idempotent, that the hedge budget has capacity, and that a different replica is available.
- • The hedge is dispatched to that other replica and the client takes whichever response arrives first.
- • The loser is cancelled, and the achieved hedge rate is fed back so the threshold keeps the rate near its target.
- • The threshold is a constant and drifts out of alignment with the real distribution, so the hedge rate rises toward 100%.
- • The hedge is routed to the same replica, or to one behind the same overloaded dependency.
- • The operation is not idempotent and duplicate side effects occur at roughly the hedge rate.
- • The cancellation of the loser is lost, so both replicas execute in full and the load cost doubles.
- • The slowness has a shared cause, so hedging adds load without improving latency at all.
- • Hedge rate at 100%: request rate to a dependency doubles during a latency regression while user traffic is flat. The threshold is stale or the budget is missing, and hedging is now the load source.
- • Duplicate side effects: two identical writes with different request ids for the same user action, at a rate suspiciously close to the hedge rate, and no errors in any log.
- • Hedging with no latency benefit: hedge rate is healthy at 5%, extra load is visible, and p99 has not moved. The cause of the slowness is shared, so both replicas are equally slow.
- • Amplified overload: an overloaded dependency gets 5–10% more traffic exactly when saturated, pushing it past a knee. Signature is p99 worsening *after* hedging was enabled, with hedge rate climbing alongside.
- • Plain hedging requires no coordination — a purely client-side decision, which is most of its appeal.
- • Tied requests require servers to exchange a drop notice, a lightweight one-way coordination that is best-effort and never load-bearing for correctness.
- • The hedge budget is per client process and uncoordinated, so fleet-wide hedge load is the per-client ratio applied to each client’s own traffic — acceptable precisely because the ratio is proportional.
- • Client-observed latency is the minimum of the attempts, so a single degraded replica becomes invisible to callers.
- • Both replicas may execute, so any operation hedged must be safe to run twice; that safety is the guarantee doing the work here.
- • When the budget is exhausted, hedging stops and behaviour degrades cleanly to the un-hedged case rather than to something worse.
- • Detect: hedge rate as a first-class metric. It should sit near the target; anything above 10% means the threshold has drifted or the system is in a shared-cause slowdown.
- • Contain: cap hedge traffic by budget and disable hedging automatically when the dependency’s error rate is elevated — hedging into a failing dependency only adds load.
- • Recover: as latency normalises the threshold rises and the hedge rate falls on its own; no operator action is required if the threshold is live.
- • Reconcile: audit for duplicate effects on any hedged operation, using the client-chosen key as the join. The hedge rate is the upper bound on how many duplicates were possible.
- • Verify: inject slowness into one replica and confirm client p99 improves; inject slowness into all replicas and confirm hedging backs off rather than piling on.
- • Hedge rate — hedges divided by requests — which is the primary control signal and should track the target closely.
- • Hedge win rate: how often the hedge beat the original. A low win rate means the threshold is too aggressive or the replicas are correlated.
- • Latency percentiles with hedging on and off, ideally as an A/B on a traffic slice, because the aggregate improvement is easy to attribute wrongly.
- • Duplicate execution rate at the server, keyed by request id — the direct measure of the load cost and of cancellation working.
- • Read paths with many interchangeable replicas, where slowness is caused by per-replica events like GC, compaction or a cold cache.
- • Fan-out systems where one slow shard dominates the aggregate, which is the situation described in Fan Out to 100 and the Component’s Tail Becomes the System’s Median.
- • Latency-sensitive user-facing paths with a strict tail objective and spare capacity to fund a few percent extra load.
- • Non-idempotent operations, where hedging is not a latency technique but a duplication mechanism.
- • Systems already near capacity, where 5% extra load at the worst moment can push a dependency past its knee.
- • When slowness is correlated across replicas — a shared dependency, a hot key, an expensive query — in which case hedging is pure cost.
- • Fix the tail at its source: GC tuning, cache warm-up on start, avoiding cold shards. Slower to achieve and strictly better when possible.
- • Reduce fan-out width so fewer components can contribute to the tail. See Fan Out to 100 and the Component’s Tail Becomes the System’s Median.
- • Return partial results at the deadline instead of waiting for stragglers — often a better user experience than a slightly faster complete answer.
- • Load-balance on measured latency or least-outstanding-requests, which avoids slow replicas before dispatch rather than compensating afterwards.
Hedged requests: buy the tail back with a little load
| Distribution | p50 | p90 | p99 | p99.9 |
|---|---|---|---|---|
| one call, no hedge | 20 ms | 104 ms | 400 ms | 1.07 s |
| one call, hedged | 20 ms | 73 ms | 137 ms | 250 ms |
| slowest of 10, no hedge | 138 ms | 391 ms | 1.07 s | 2.40 s |
| slowest of 10, hedged | 81 ms | 135 ms | 250 ms | 431 ms |
What people believe, and what is true
Hedging is just a retry sent early.
A retry follows evidence that the first attempt may not have landed. A hedge is sent while the first attempt is healthy and in flight, so both executing is the expected case rather than the unlucky one.
Hedging doubles the load.
Only if the threshold is wrong. At the p95 it adds 5% by construction — the cost is set by the percentile you choose.
Hedging helps when the service is overloaded.
It adds load when the service is least able to take it, and the slowness is shared so the hedge is equally slow. Hedging should back off under elevated errors, not lean in.
A fixed 200ms hedge threshold is fine.
It is fine until p95 crosses 200ms, at which point every request hedges and load doubles during the exact regression you were protecting against.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
If a request is taking longer than 95% of requests normally do, ask a second replica and use whichever answers first. That costs about 5% more requests and removes most of the tail.
Practical
Track p95 live, hedge only idempotent operations, hedge to a different replica, cap hedge traffic with a budget, cancel the loser, and disable hedging when the dependency’s error rate rises. Alert on hedge rate: it should hover near the target and never climb.
Advanced
Use tied requests where you own both ends: enqueue on two replicas with mutual identity and have the first to dequeue cancel the other, so queueing delay is minimised while duplicate execution is limited to the cancellation window. Then confront the independence assumption directly — measure the joint distribution of replica latencies rather than assuming a product. Where slowness is correlated, hedging is a pure cost, and knowing which regime you are in is the difference between a technique and a superstition.
Apply it
- 💬 Why does hedging at the p95 cost only about 5% extra load?
- 💬 Why is idempotence a stricter requirement for hedging than for retries?
- 💬 Your hedge rate has climbed to 60% during a latency regression. What went wrong and what do you do?