What Counts as the Same Operation?

The receiver cannot observe that two requests are the same operation. It can only observe that they carry the same token, and then *decide* to treat that as identity. Everything difficult here follows from the gap between those two things — because the caller and the receiver are different machines with different beliefs about what the token means.

The caller believes: "this key names the operation I am attempting; my retries carry it; nobody else uses it." The receiver believes: "this key names an operation I may already have applied; if I have, I return what I returned before." Those two beliefs only line up if there is an agreement, made at design time, about the key’s namespace, its lifetime, and what happens when the same key arrives with different content. Nothing in the protocol enforces that agreement, which is why it fails silently.

This is the reason this lesson is not the same as the mechanism. Storing a key and replaying a response is a small amount of code. Deciding what the key *means* across two organisations, three regions and a retention policy is a distributed-systems design problem, and it is where the real bugs live.

A key is a string a client chose. Clients choose badly. They use 1, test, order-1, an auto-incrementing integer, or a UUID that a badly-seeded random generator repeats. Two different customers of your API will collide, and when they do, an unscoped dedup store returns one tenant’s stored response to another tenant’s request.

That is not a correctness bug in the ordinary sense — it is a data disclosure. The stored response may contain an order id, an amount, an address. The remedy is trivial and must be non-optional: the dedup record is keyed by (tenant_id, api_key_id, endpoint, client_key), never by client_key alone, with the tenant taken from the authenticated principal and never from the request body (Security owns the general form of this rule under tenant isolation).

The endpoint component deserves its own thought. If a client reuses one key across POST /charges and POST /refunds, should the second return the first’s response? Almost certainly not — but "almost" is doing work, because some designs deliberately scope a key to a whole logical operation spanning several endpoints. Decide explicitly and document it, because the client cannot infer it.

A client sends key K with a €50 charge. It later sends key K with a €500 charge — a bug in their retry loop, a reused variable, a form resubmitted after an edit. Three policies exist and only one is safe.

Ignore the body and replay the first response. The client believes it charged €500. You charged €50 and told them it worked. This is the worst option because it is silent and the divergence is between your record and their belief, which nothing will reconcile.

Treat it as a new operation. Then the key means nothing, and a genuine retry with a body that differs by a timestamp field will double-charge. Also silent.

Reject it. Store a fingerprint of the canonicalised request body with the record; if a request arrives with a matching key and a non-matching fingerprint, return a definite error (Stripe returns a 400 with idempotency_key_in_use semantics; the exact code matters less than that it is a 4xx and not a replay). The client learns immediately that it has a bug, and no wrong effect occurs. This is the only option where the failure is loud.

Canonicalisation is the subtle part: JSON key order, absent versus null fields, numeric formatting and a client-side timestamp in the body will all change the hash without changing the operation. Fingerprint a canonical form and exclude fields that legitimately vary between attempts, or you will reject correct retries — which teaches clients to generate a fresh key per attempt, which destroys the mechanism entirely.

The dedup store cannot grow forever. Records must expire, and the moment a record expires, a request carrying that key becomes a new operation. So the guarantee is not "at most once" — it is "at most once within the retention window", and the window must be chosen against the maximum possible retry horizon, which is almost always longer than anyone estimates.

The client’s own retry loop might last thirty seconds. But a message sitting in a dead-letter queue may be replayed by an operator on Monday morning after failing on Friday night. A webhook sender may retry with exponential backoff for three days. An operator may click "re-drive" in an admin console a week later. A batch job may be re-run from a checkpoint. Each of those presents the original key long after any client-side retry would have given up.

So the rule is: retention ≥ the longest path by which the same key can legitimately reach you again, plus margin. Twenty-four hours is a common default and is too short for any system with a manual DLQ workflow. And because retention is finite, expiry must be observable: a metric for eviction rate and for the age of the oldest retained record, because an eviction is a future duplicate you have already accepted.

The storage cost is real and is the honest counter-pressure. A high-throughput API storing a fingerprint and a response body per operation for seven days is storing a lot. The usual compromise is to keep the *claim* (key plus fingerprint plus status) for a long window and the *response body* for a short one — a replay after the body expires can then return a definite "this operation was already applied" rather than the original payload, which is far better than treating it as new.

The dedup store is itself a distributed system, and its topology decides whether the guarantee survives the events that produce the most retries.

Co-location with the effect. If the claim and the effect commit in one transaction in one store, the claim is exact. If the claim lives in Redis and the effect in Postgres, there is a window: crash after claiming and before committing, and the operation is now permanently blocked (the claim says done, the effect never happened); crash after committing and before claiming, and the retry duplicates. Which of the two you get is decided by the order, and neither is good. Separate stores buy latency and cost correctness.

Partitioning. The dedup store is partitioned by something. If the retry routes to a different partition than the original — because the partition function includes a field that changed, or because the routing layer rebalanced — the check finds nothing. Dedup state must be partitioned by the dedup key itself, so every attempt with that key lands on the same shard (Hash Partitioning and the Modulo Trap).

Failure domain. This is the sharp one. A regional failover is exactly the event that produces a burst of retries — clients time out, retry, and get routed to region B. If the dedup store is regional and not replicated synchronously, region B has never seen any of those keys. Every in-flight operation duplicates, precisely at the moment the system is least able to cope. The dedup record must be in the same failure domain as the effect: if the effect fails over, the claim must fail over with it, atomically. A globally replicated dedup store with asynchronous replication is worse than useless here, because the replication lag is measured in exactly the window the retries occupy (EU and US Are Partitioned. Can Both Keep Accepting Writes?, Three Ways to Accept a Write in More Than One Place).

A claim has a status, not just an existence. Three cases behave differently and conflating them causes real incidents.

In progress. The first attempt is still running when the retry arrives. Returning "already applied" is wrong — nothing has been applied yet. The correct answer is a definite "in progress, retry later" (a 409, or a 425), because the caller must not conclude success and must not start a third attempt that races the first two.

Completed. Replay the stored result. This is the case everyone implements.

Failed. Now it matters *why*. A deterministic failure — validation error, insufficient funds, unknown account — will fail identically on every retry, and caching it saves work and gives the client a consistent answer. A transient failure — a downstream timeout, a lock conflict, a 503 — must not be cached, because caching it burns the key: every retry replays the failure and the client can never succeed with that operation identity, even though the operation is perfectly valid. The client’s only escape is to generate a new key, which is precisely the behaviour that reintroduces duplicates.

So the rule is: cache deterministic failures, release the claim on transient ones. Getting this wrong produces a support case that reads "your API returns the same error forever for this order", and the cause is three layers away from where anyone looks.