Retries Are a Property of the Operation
Retry safety is not something a caller can decide. It is a fact about the operation, and it has to be stated in the interface — otherwise every caller is guessing, and some of them will guess wrong.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind survives until the requirement changes.
What must be true about this operation before anyone is allowed to call it twice?
A shared HTTP client was given "retry three times on 5xx" as a global default. A month later, some customers have three identical orders, and the client library is doing exactly what it was configured to do.
Configure the HTTP client to retry on 5xx and timeouts. It is one line, it applies everywhere, and it demonstrably reduces error rates in the dashboard.
It applies a caller-side policy to an operation-side property. The client cannot know whether POST /orders is safe to repeat, so the policy is correct for the reads and wrong for exactly the writes where being wrong is expensive.
- It applies a caller-side policy to an operation-side property. The client cannot know whether
POST /ordersis safe to repeat, so the policy is correct for the reads and wrong for exactly the writes where being wrong is expensive. - The error rate genuinely drops, which is the trap: the dashboard improves while duplicate orders accumulate somewhere nobody is looking.
- Under a real outage, every caller retries simultaneously and the retry traffic is several times the original load — arriving precisely when the dependency is least able to serve it (Retry Storms: The Load You Generated Yourself in Backend).
- Retries compose multiplicatively through layers. Three tiers each retrying three times is twenty-seven requests for one user action, and no layer can see the total (One Retry per Tier Is Not One Retry — It Multiplies in Distributed Systems).
What limits the solution, and what must never stop being true
This domain leads with these two. A design that ignores its constraints is not a design, and an invariant nobody named is one nothing is protecting.
- Retries will be added by people who did not write the operation — a client library, a proxy, a queue with redelivery, a job runner (What Changes at the Network Boundary).
- The operation is one of two hundred behind the same client, so a per-operation decision has to be discoverable without reading each implementation.
- The downstream service has finite capacity, so retries under load are a load multiplier as well as a correctness question.
- An operation is retried only if repeating it produces no additional effect, or if the caller has explicitly accepted that it might (Idempotency by Design).
- Total work sent to a dependency must be bounded, whatever the failure rate. Unbounded retry converts a degraded dependency into a dead one (One Retry per Tier Is Not One Retry — It Multiplies in Distributed Systems).
- A retry never outlives the caller's deadline. Retrying after the client has given up is pure load with no possible benefit.
Who owns what, and where the seams fall
Responsibilities decide boundaries; boundaries decide what an interface has to say.
- The operation owns declaring whether it is safe to retry, and under what condition. That declaration belongs in the type or the interface, not in a wiki page (Designing a Module Interface).
- The caller owns the policy — how many, how long, with what backoff — within the safety the operation declared.
- The system owns the budget. Someone must own the fact that total retry traffic is bounded across all callers, and no individual caller can own that (Cap Retries as a Fraction of Traffic, Not as a Count per Request in Distributed Systems).
- The failure classification owns the distinction between retryable and permanent, because retrying a permanent failure is guaranteed waste (An Error Taxonomy That Survives Contact).
- The safety declaration lives at the operation boundary; the policy lives at the call site; the budget lives at the process or fleet boundary. Three different scopes, and collapsing them is how the global default happened.
- The retry boundary should be the outermost place with enough context to decide — usually one layer, not every layer. Nested retries at three levels is the failure mode (Fan-in and Fan-out).
- Deadlines cross the boundary with the call, so a retry policy that ignores the remaining budget is retrying into a void (Pass the Remaining Budget Down, Not a Fresh One in Distributed Systems).
Before you retry, three questions
The decision is not "how many attempts". It is three prior questions, and if any of them is unanswered the attempt count is a guess with a number on it.
The middle question is the one that produces the duplicate orders, and it is the one a generic client cannot answer.
Do you know the work did not happen, or is repeating it harmless?
when Connection refused, DNS failure, a 400, a request rejected before dispatch.
cost Retry freely; safety is not in question. Cost: only load, so backoff and a budget are all you need (Without Jitter, Every Client That Failed Together Retries Together in Backend).
when A timeout on createPayment(commandId, ...).
cost Retry with the same id. This is the target state, and getting operations here is more valuable than any retry policy (Idempotency by Design).
when A timeout on POST /orders with no command id.
cost Do not retry automatically. Record the unknown, reconcile, and treat the missing id as the design bug it is (Designing for Failure).
when A 4xx, a validation error, a poison message.
cost Never retry. Fail fast and route it somewhere a human can see (A Dead-Letter Queue Is a Workflow, Not a Bin in Backend).
when Rising latency, timeouts under load, a 429.
cost Retrying is actively harmful — shed load and back off hard. This is where retry policy causes the outage it is meant to survive (Rejecting Work on Purpose — and Rejecting It Cheaply Enough to Help in Distributed Systems).
Say it in the interface
Documentation is the wrong place for retry safety, because the thing that decides is a generic client that does not read documentation. Put it where a client can read it — a type, an attribute, a convention — and the correct default becomes the automatic one.
The version below is deliberately unremarkable. The point is not the mechanism but that a caller with no knowledge of payments can now behave correctly.
1type Retry =2 | { safe: 'always' } // read, or naturally idempotent3 | { safe: 'with-id' } // needs a command id4 | { safe: 'never'; why: string } // and the reason, for the reader5 6interface Operation<Req, Res> {7 name: string8 retry: Retry9 call(req: Req, deadline: Deadline): Promise<Res>10}11 12const createPayment: Operation<CreatePayment, Payment> = {13 name: 'createPayment',14 retry: { safe: 'with-id' },15 call: (req, d) => post('/payments', req, d),16}17 18// the generic client no longer has to guess19async function invoke<Q, R>(op: Operation<Q, R>, req: Q, d: Deadline) {20 const attempts =21 op.retry.safe === 'never' ? 1 :22 op.retry.safe === 'with-id' && !hasCommandId(req) ? 1 : 323 return withBackoff(attempts, d, budget, () => op.call(req, d))24}Two things are doing the work. The with-id case degrades safely — if a caller forgot the command id, the client refuses to retry rather than retrying unsafely. And never carries a reason, so the next person to consider changing it argues with the reason instead of with the annotation (Architecture Decision Records).
The retry that causes the outage
The most damaging retry bug is not a duplicate. It is the one where retries are the reason a degraded dependency never recovers — and it looks like diligent engineering right up to the incident.
looks like A global client policy of three attempts on any 5xx or timeout, applied to every operation; nested retries at the gateway, the service and the worker; no budget, no jitter, and a dashboard showing the error rate improved when it was introduced.
suggests Retries are being used to paper over a dependency that is chronically degraded, so the underlying problem is invisible and the load multiplier is armed. Under a real outage the dependency receives several times its normal traffic at its weakest moment, and each layer's retries multiply the next (Cascading Failure: When the Response to Failure Causes More Failure in Distributed Systems).
fix Separate the three concerns: declare safety on the operation, set policy at one layer, and enforce a budget across the fleet. Then check the number nobody has checked — how many requests one user action produces when everything downstream is failing (Cap Retries as a Fraction of Traffic, Not as a Count per Request in Distributed Systems).
How to build it
Most important first.
- State retry safety in the interface. A marker on the operation — a type, an attribute, a naming convention — that a generic client can read is worth more than any documentation, because the client is what actually decides (Units in Names and Types).
- Make the unsafe operations safe rather than forbidding retries: add the command id and the question disappears. That is the better fix and it is available more often than people think (Idempotency by Design).
- Classify failures before retrying. Connection-refused is safe and worth retrying; a 400 is permanent; a timeout is ambiguous and only retryable if the operation is idempotent (Designing for Failure).
- Retry at one layer. Pick the layer with the most context — usually the one that knows the user's deadline — and make the others fail fast (Error Boundaries).
- Bound the total, not the attempts. A retry budget expressed as a fraction of successful traffic degrades gracefully under a full outage; "three attempts each" triples load exactly when that is fatal.
- Use backoff with jitter, and derive the deadline from the caller's remaining budget rather than from a constant (Without Jitter, Every Client That Failed Together Retries Together in Backend).
- Stop when the deadline is gone. A retry that will complete after the caller has given up is load with no upside (Rejecting Work on Purpose — and Rejecting It Cheaply Enough to Help in Distributed Systems).
What the next change costs
The field this whole domain exists for. A structure is only better if it makes the change after this one cheaper — and it is worth saying which changes it does not help.
- With safety declared in the interface, "we now retry writes too" costs one policy change at one call site, because the client can already tell which operations permit it. Minutes.
- Without it, the same change costs an audit of two hundred operations to determine which are safe, and the audit has to be redone whenever an operation changes. The cost of the next retry-policy change is proportional to the size of the API, which is the shape to avoid.
- Adding a new operation costs one declaration under the designed approach and, under the naive one, costs nothing — and silently inherits a policy that may be wrong for it. Cheap now, expensive later, which is the standard shape of this kind of debt (Accidental Debt).
- Declaring safety in the interface is more ceremony on every operation, including the many for which it is obvious.
- Retrying at one layer means an inner transient failure surfaces further out and looks worse in inner-layer metrics, which makes some teams uncomfortable and is nonetheless correct.
- Budgets mean that during a partial outage some requests fail that a retry would have rescued. That is the trade — individual success rate for systemic survival — and it should be made explicitly.
What can go wrong
- Retries are added and idempotency is not, so the mitigation creates the duplicates. This is the most common sequence and it is worth naming explicitly: retries without idempotency are a decision to sometimes do the work twice.
- Retry storms: synchronised retries after a blip keep the dependency down. The fix is jitter and a budget, and the failure of the fix is a budget nobody monitors.
- Nested retries multiply invisibly, and the total is discovered only from the dependency's traffic graph during an incident.
- A permanent failure is retried forever — a poison message that consumes a worker indefinitely because nobody classified it (A Dead-Letter Queue Is a Workflow, Not a Bin in Backend).
- Retries hide a chronically degraded dependency until it fails completely, so the first real signal is the outage rather than the degradation (Debuggability by Design).
- Callers depend on the safety declaration, which means it must be part of the published interface rather than an implementation detail (API Stability).
- The retry layer depends on failure classification being accurate, so a dependency that returns 500 for validation errors makes correct retry impossible — an argument for fixing status codes upstream (An Error Taxonomy That Survives Contact).
- A shared retry budget is a coordination dependency across callers, which is genuine coupling accepted deliberately to protect a shared resource.
- "Retries improve reliability." They improve it for transient, independent failures. For a saturated dependency they are the thing making it worse (Retry Storms: The Load You Generated Yourself in Backend).
- "Idempotent means retryable." Necessary, not sufficient: retrying an idempotent operation into an overloaded dependency is still a load multiplier (Rejecting Work on Purpose — and Rejecting It Cheaply Enough to Help in Distributed Systems).
- "Exponential backoff solves storms." Without jitter, backed-off retries stay synchronised and arrive in waves (Without Jitter, Every Client That Failed Together Retries Together in Backend).
- "The client library handles retries." It handles the mechanics. It cannot know which of your operations are safe, and that is the part that matters.
- boolean-parameters
Testing it, and how it ages
- For every operation declared retry-safe, a test that calls it twice and asserts one effect. The declaration must be verified or it is a comment (Property-Based Testing).
- A test that a permanent failure is not retried, asserting attempt count — otherwise a misclassification burns quota silently.
- A load test with the dependency failing, asserting that total request volume stays inside the budget (Stress Testing: A Test That Passed Once Proves Nothing in Concurrency).
- A test that retries stop at the deadline rather than at the attempt count.
- Retry policy tends to accumulate at every layer as each team adds its own. Periodically counting the total attempts for one user action is worth doing; the number is usually a surprise.
- As the system grows, the budget becomes the important control and the per-call attempt count becomes almost irrelevant.
- Once most operations carry command ids, the safety question largely disappears and the conversation moves entirely to load — which is the healthy end state (Idempotency by Design).
Where this applies
This domain's advice is contested more than most. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view rather than a caricature.
- GENERALThat whether repeating an operation is safe depends on the operation rather than on the caller is definitional, so it holds for HTTP, RPC, queue consumers and in-process calls alike.
- SCALE-SPECIFICFor a single client and a dependency with headroom, naive retries are genuinely fine and the budget machinery is over-engineering. Above roughly a few hundred requests per second, or wherever many callers share one dependency, retry amplification becomes the dominant failure mode and the same defaults become dangerous.
- CONTESTEDA defensible opposing view holds that per-operation safety declarations are process that decays — the annotation drifts from the implementation, nobody updates it, and it becomes a lie that is trusted. That camp argues for one blunt rule instead: make every mutating operation idempotent, then retry everything uniformly and stop reasoning about it. That is a genuinely good design when you control every operation, and it is unavailable the moment a third-party API is in the path — which is most of the time, and is why the declaration still has to exist somewhere.
Where the depth lives
This domain teaches the codebase-level structure and hands the rest off.
- — Testing & Reliability Engineering — measuring retry amplification in production, running the load test that proves a budget holds, and the alerting that catches a retry storm early belong there.
- — System Design — circuit breakers, bulkheads and capacity planning for retry traffic are system-level responses to the same problem; this lesson stops at what the operation has to declare.