The question this answers
The outcome is unknown. Should I retry — and what have I actually done to the receiver when I do?
Retrying converts "unknown outcome" into at-least-once execution, provided the caller keeps retrying until it gets a definite answer. Not retrying converts it into at-most-once execution. Neither converts it into "exactly once", and no policy tuning changes that. The only thing under your control is which error you would rather have.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The caller knows how many attempts *it* has sent, and nothing about how many *executed*. Those numbers are unrelated: three attempts may have produced zero, one, two or three effects. The receiver knows how many requests it received but cannot tell a retry from a new request unless the request itself carries an identity the caller chose — and any identity the *receiver* generates is useless here, because the case that matters is precisely the one where the caller never saw the response.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Two errors, and you pick which one to have
A Timeout Tells You Nothing About Whether It Happened establishes that after a timeout you cannot learn what happened. What follows is that "should I retry?" is not a question about the world; it is a question about cost. Retry and you risk a duplicate effect. Do not retry and you risk a lost effect. There is no third branch, and no amount of instrumentation moves you off the fork.
So the design question is a comparison. What does a duplicate cost, and what does a loss cost? A duplicate charge costs a refund, a support contact, and some trust. A lost charge costs the whole order and a customer who thinks they bought something. For most commerce operations the loss is worse, which is why at-least-once is the industry default. Invert the costs and the answer inverts: a duplicate "unlock the door" command may be harmless while a duplicate "fire the actuator" is not.
Write the comparison down explicitly for each operation class. Teams that skip it end up with a single global retry policy applied to operations whose cost asymmetries point in opposite directions, and then discover the mismatch in an incident.
| Operation | Cost of a duplicate | Cost of a loss | Therefore |
|---|---|---|---|
| Charge a cardtypical | Refund + support contact | Unpaid order, lost revenue | Retry; make it idempotent |
| Send a marketing emailtypical | Annoyed recipient, unsubscribe | One fewer email | Do not retry blindly |
| Append to an audit logtypical | A duplicate line, dedupable later | A compliance gap | Retry aggressively |
| Emit a metric sampletypical | Skewed aggregate | One missing point in thousands | Do not retry |
| Dispatch a physical shipmentassumption | A second parcel and a recall | Nothing ships | Retry only with strong dedup |
Identity must exist before the first attempt
This is the point where retries stop being a client concern and become a protocol design concern. From the receiver’s side, your retry is simply another request. It has a new connection, a new timestamp, possibly a different server. Nothing about it says "I am the same operation as one you may already have processed" — unless the request carries an identifier that is stable across attempts.
And that identifier must be generated by the caller, before the first attempt is sent. The tempting alternative — let the server assign an id and return it — fails in exactly the case that matters. If you never received the response, you never received the id, so your retry cannot reference it. A server-assigned identity resolves duplicates for every scenario except the one that produces duplicates.
This is why POST /orders with a client-generated key behaves so differently from POST /orders without one, and it is the reason idempotency keys look the way they do everywhere they appear. The mechanism — headers, storage, response replay — belongs to API Design and to your service framework. What belongs here is the reason the identity has to originate at the caller, which is a pure consequence of the ambiguity.
1// Broken: the identity exists only if the response arrived — and the2// case we care about is precisely the one where it did not.3const res = await post('/charges', { amount }) // times out4const id = res?.id // undefined5await post('/charges', { amount }) // a SECOND charge6 7// Correct: identity is created before anything leaves the process, and8// is the same on attempt 1 and attempt 7.9const key = crypto.randomUUID()10for (const attempt of attempts()) {11 try {12 return await post('/charges', { amount }, { idempotencyKey: key })13 } catch (e) {14 if (!isRetryable(e)) throw e // a definite 4xx is an answer, not silence15 await sleep(backoffWithJitter(attempt))16 }17}18// Note what the key does NOT do: it does not tell you whether the first19// attempt succeeded. It makes not knowing harmless.The retry can overtake the original
The mental model most people carry is sequential: attempt one finishes (or dies), then attempt two starts. The network does not work that way. Attempt one may be sitting in a buffer, or executing slowly on a server you have given up on, while attempt two is already running elsewhere. Both are live. Both may commit.
Two consequences follow, and both are counter-intuitive. First, duplicate execution can be concurrent rather than sequential, so a dedup check of the form "look up the key; if absent, do the work; then record the key" is a race: both attempts look up, both find nothing, both proceed. The check and the record must be one atomic operation — a conditional insert, a unique constraint, a compare-and-set — in the same store as the effect.
Second, the later attempt can commit before the earlier one, which means the original can land after you have already decided it failed and moved on. If your retry updated a record and the delayed original then overwrites it with the older body, you have a stale write with no error anywhere. This is the same zombie-write shape that makes compensations dangerous (A Refund Is Not a Rollback), and it is why fencing or version checks matter even for plain retries (Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely).
When you may not retry
Retrying is the default, not a law. Several conditions make it wrong, and each one is a real production incident when ignored.
A definite answer is not silence. A 400, 422 or 403 is information: the receiver reached a decision and reports it. Retrying that wastes capacity and, if the response was a validation failure, will never succeed. Split retryable from non-retryable at the response class, not at "did I get an exception".
The deadline is not yours alone. If the caller’s caller has 200ms of budget left, spending 500ms on retries produces work whose result nobody will read. Deadlines must propagate, and a retry that cannot finish within the remaining budget must not be attempted (Pass the Remaining Budget Down, Not a Fresh One, A Deadline Is Divided Across the Call Chain, Not Repeated at Every Hop).
Retries compose multiplicatively. Three layers each retrying three times is up to 27 requests for one logical call, and each layer thinks it is being modest. Choose one layer to own retries — usually the one closest to the ambiguity, and never every layer. And retries are load exactly when the dependency is least able to take it, which is how a slow service becomes a dead one (One Retry per Tier Is Not One Retry — It Multiplies, Cap Retries as a Fraction of Traffic, Not as a Count per Request). Backoff with jitter and a retry budget expressed as a fraction of base traffic are the standard containment (Without Jitter, Every Client That Failed Together Retries Together).
- Definite failure — a 4xx is an answer. Retry only ambiguity and transient errors.
- Budget exhausted — no attempt whose result would arrive after the caller’s deadline.
- One retrying layer — pick it deliberately; disable retries in the others.
- Bounded amplification — cap retries as a percentage of base traffic, not per-request.
- Jittered backoff — synchronised retries turn a blip into a thundering herd.
- Non-idempotent and undedupable — if you cannot make a duplicate safe and cannot detect it, the honest answer may be to fail and escalate rather than retry.
Key points
- After an ambiguous outcome, retrying gives at-least-once and not retrying gives at-most-once. There is no third option.
- Which one you want is decided by the cost asymmetry between a duplicate and a loss, per operation class.
- A retry is indistinguishable from a new request unless the caller generated a stable identity before the first attempt.
- A server-generated id cannot help, because the failure case is precisely the one where the response never arrived.
- Attempts overlap: duplicate execution can be concurrent, so the dedup check must be atomic with the effect.
- A delayed original can commit after the retry, producing a stale write with no error.
- Do not retry definite failures, do not retry past the propagated deadline, and do not retry at every layer.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The caller generates a stable operation identity before its first attempt and keeps it for the life of the operation.
- • Attempt one is sent with a deadline. It ends in success, a definite failure, or silence.
- • On silence or a transient error, the caller waits a jittered backoff interval and re-sends — the same identity, the same body.
- • The receiver attempts an atomic claim on the identity in the same store as the effect: insert-if-absent, or a unique constraint.
- • If the claim succeeds, the work runs and its result is recorded against the identity. If it fails, the recorded result is returned.
- • The caller stops on a definite answer, on budget exhaustion, or on the propagated deadline — and records which, because those are different outcomes.
- • Both attempts are in flight simultaneously and both pass a non-atomic dedup check.
- • The delayed original commits after the retry, overwriting newer state.
- • The retry carries a different body from the original, so identity and content disagree (What Counts as the Same Operation?).
- • Retries at several layers multiply into an order of magnitude more load than intended.
- • Synchronised backoff across many callers produces a coordinated retry wave.
- • The retry succeeds but its response is also lost, so the caller retries again — an unbounded loop against a working server.
- • The operation is retried after the caller’s deadline passed, producing an effect nobody is waiting for.
- • Duplicate effects at a rate that tracks dependency latency: the operator sees duplicate charges rise whenever the payment provider’s p99 crosses the client timeout, with no error rate change at either end.
- • Load amplification during a partial outage: request rate to a degraded dependency rises 5–10× while user traffic is flat. The dependency’s recovery is prevented by its callers.
- • Stale overwrite with no error: a record shows an older value after an update that the application logged as successful. The cause is a delayed original landing after its retry, and nothing in either log shows an anomaly.
- • Retry storms synchronised to a deploy or a scheduled job, visible as periodic spikes at exactly the retry interval — a signature of missing jitter.
- • Wasted capacity on definite failures: a validation error retried three times per request, tripling load on a code path that can never succeed, and inflating the error rate metric threefold.
- • The retry itself needs no coordination; making it safe does. The atomic claim on the identity is a coordination point, and its availability becomes part of the operation’s availability.
- • Putting the claim in the same store as the effect makes it free — it rides an existing transaction. Putting it in a separate store adds a network hop and a window in which the two disagree.
- • Retry budgets are coordination across callers: a per-request retry count coordinates nothing, while a fleet-wide budget bounds the total load a dependency can be subjected to.
- • Deadline propagation is coordination across the call graph, ensuring nobody spends effort on work whose result has already been abandoned.
- • The receiver’s own consistency is unaffected; each executed attempt is a correct local transaction.
- • The relationship between caller intent and receiver effect is what breaks — one intent, zero or several effects.
- • A caller that stops retrying before a definite answer leaves an operation in a permanently unknown state, which nothing will resolve on its own.
- • Under load, retries make the dependency’s failure worse, so the failure mode is self-reinforcing rather than self-limiting.
- • Detect: measure the ambiguous-outcome rate (timeouts) separately from the error rate, and the duplicate-claim rate at the receiver. Both are normally non-zero and normally stable.
- • Contain: enforce a retry budget as a fraction of base traffic and shed retries before shedding first attempts, so a struggling dependency sees load fall rather than rise.
- • Recover: re-drive operations whose outcome was never resolved, using the same identity so a duplicate collapses into the original.
- • Reconcile: compare caller-side intent records with receiver-side effect records on the shared identity; the delta is exactly the population of unresolved operations.
- • Verify: confirm that duplicate rate returns to baseline and that no operation remains in an unknown state older than the retry window.
- • Timeout rate as its own metric, never merged into the error rate — they call for opposite responses.
- • Attempts per logical operation, p50 and p99. A p99 that climbs is amplification starting.
- • Duplicate-claim rate at the receiver, keyed by operation identity: the direct measurement of how often retries are saving you.
- • Ratio of retry traffic to first-attempt traffic per dependency — the early warning for a retry storm.
- • Operations abandoned without a definite answer, which is the population that reconciliation must handle.
- • Retry inter-arrival distribution; a sharp spike at a fixed interval means jitter is missing somewhere.
- • Whenever the failure is transient — a restarted pod, a leader election, a brief network blip — where a second attempt genuinely succeeds.
- • Where a duplicate is cheap or detectable and a loss is expensive, which describes most write paths in commerce.
- • Where the receiver already offers an atomic claim on an identity, so the safety costs nothing extra.
- • Where the operation is naturally idempotent, in which case retries are free and the whole analysis collapses (Idempotent Is a Property of the Whole Effect, Not the Write).
- • Against a dependency that is failing because it is overloaded — every retry deepens the hole (One Retry per Tier Is Not One Retry — It Multiplies).
- • For operations with irreversible, undedupable effects, where a duplicate is worse than a loss.
- • When applied to definite failures, wasting capacity on requests that can never succeed.
- • When layered, so that a modest policy at each level composes into an aggressive one overall.
- • When the caller is a user-facing request with a tight deadline; retrying past the point where the user has left is pure waste.
- • Make the operation naturally idempotent so a duplicate is harmless and the decision stops mattering.
- • Hand the operation to a durable queue and let a consumer own the retrying, with the queue providing the durability and the backoff (Work Queues: One Task, One Worker, Competing Consumers).
- • Fail fast and reconcile later, where an asynchronous repair is cheaper than the machinery to make retries safe.
- • Hedge instead of retry for read-only work: send a second request early rather than after a timeout, and take the first answer (Send a Second Request After p95 and Take Whichever Answers First).
- • Return a durable operation handle to the caller and let it poll, converting an ambiguous synchronous call into an unambiguous asynchronous one.
The retry is a decision, not a reflex
What people believe, and what is true
Retrying is safe because the first attempt failed.
The first attempt had an unknown outcome. That is the entire premise. A retry after ambiguity is a possible duplicate by construction.
The server can just detect duplicates by comparing request contents.
Two genuinely different orders for the same item at the same price are byte-identical. Content is not identity; the caller must supply identity.
Attempts happen one after another, so the second sees the first’s effect.
They overlap. The original may still be executing, and may commit after the retry does. Concurrency is the normal case, not the edge case.
An idempotency key makes retries safe.
Only if the receiver claims it atomically with the effect. A read-then-write check lets two concurrent attempts both pass.
More retries mean higher reliability.
Beyond a small number they mean higher load on the thing that is failing. Reliability comes from the first retry; everything after is mostly amplification.
Exponential backoff solves retry storms.
Only with jitter. Without it, all callers back off in lockstep and retry simultaneously — the storm arrives on a schedule.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
You cannot learn what happened, so you choose which mistake to make: a possible duplicate, or a possible loss. Retry when a loss is worse, and give the receiver a way to recognise the retry.
Practical
Generate the operation identity in the caller before the first attempt. Retry only ambiguity and transient errors, never definite failures. Respect the propagated deadline. Retry at exactly one layer. Use exponential backoff with jitter and cap retries as a fraction of base traffic. Chart timeouts separately from errors.
Advanced
Treat the retry as creating a small set of concurrent executions of one logical operation. That framing makes two requirements obvious: the receiver’s claim on the identity must be atomic with the effect (a unique constraint or conditional insert, not a lookup), and any write must be guarded against a delayed earlier attempt landing later — a version check or a fencing token, since the ordering of your own attempts is not something the network preserves.
Apply it
- 🔧 Implement a dedup check as read-then-write, fire two concurrent retries with the same key, and reproduce the double effect. Then replace it with a unique constraint and re-run.
- 🔧 Simulate a dependency whose latency crosses your client timeout and chart duplicate rate against latency. Confirm the correlation.
- 🔧 Remove jitter from a fleet of clients and observe the retry wave; add it back and measure the difference in peak load.
- ⚡ Duplicate charges appear at 0.4% during peak hours and near zero overnight. No errors are logged. Explain the mechanism.
- ⚡ A dependency recovers from a restart, but request volume stays 6× baseline for ten minutes afterwards. What is happening and how do you stop it?
- 💬 A write times out. Walk me through how you decide whether to retry.
- 💬 Why can the server not generate the idempotency key?
- 💬 Two retries of the same request arrive at two different servers at the same instant, both carrying the same key. What must the receiver do to stay correct?
- 💬 Your service retries three times, its client retries three times, and the load balancer retries once. What is the worst case, and what would you change?
- 💬 When is a duplicate worse than a loss? Give a concrete operation.