IdempotencyGENERALPROTOCOL-SPECIFICSCALE-SPECIFIC

Idempotency Keys

A client-generated identifier that is stable across retries of one intent and unique across different intents.

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.

The question

What exactly should the client send so the server can tell a retry apart from a second, genuine request?

The requirement

Our public payments API must let clients retry safely after a timeout. We need to specify what they send and what we promise in return.

The obvious build

Hash the request body. Identical bodies are retries; different bodies are different requests. The client does not have to do anything and it works automatically.

Why it breaks

A customer legitimately buys the same coffee twice in five minutes. Identical body, two intents, and the second is silently swallowed as a duplicate. You have replaced double-charging with not-charging, which is worse because it is silent.

How it breaks in production

A customer legitimately buys the same coffee twice in five minutes. Identical body, two intents, and the second is silently swallowed as a duplicate. You have replaced double-charging with not-charging, which is worse because it is silent.
Any request containing a timestamp, a nonce, a retry_count or a client-generated request id has a different body on each attempt, so the hash differs and the duplicate is not detected at all.
JSON serialization differences between client versions change the hash for the same intent — key order, float formatting, optional fields omitted versus null.
The hash has no expiry semantics: how long is a body "the same request"? A minute? A day? Without a client-supplied identity there is no principled answer, only an arbitrary window.
The server cannot tell whether the client considers this a retry. Only the client knows whether it is retrying, and the hash throws that information away.

Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale

What is actually happening

An idempotency key is a client-generated, opaque, unique-per-intent string, held stable across every retry of that intent and never reused for a different one. The client generates it before the first attempt and reuses it for every retry of that attempt.
The direction matters: it is generated by the client because only the client knows that two requests express one intent. A server-generated identifier arrives too late — the client already had to send something.
A UUIDv4 is the standard choice: collision probability is negligible and it requires no coordination. Any opaque string with the same properties works; the server should treat it as bytes with a length limit and no structure.
The key travels in a header — Idempotency-Key is the conventional name, standardised in an IETF draft and used by most payment APIs — rather than the body, so it is not part of the payload being signed, validated or versioned.
The key identifies an attempt at an intent, not a resource. Two different keys for the same resource are two intents; one key for two different bodies is a client bug, which the server should detect and refuse (The Idempotency Key Flow).
Requests you make outbound need keys too, generated by you. Your retry of a charge call is the same problem viewed from the other side (Calling Something You Do Not Control).

The key identifies an intent, not a request and not a resource

Almost every implementation mistake here is a category error about what the key names. Name a request and it changes on every retry, defeating the mechanism. Name a resource and it stays constant across genuinely different operations, swallowing them.

It names an intent: one press of the button, one decision to send that message, one instruction to make that transfer. That is why generation belongs at the moment intent is formed — in the UI handler, in the job that decides to call the API — and not in the HTTP layer, which sees only attempts.

The practical consequence is a rule the client must follow and only the client can follow: generate once, store it with the pending operation, reuse it for every attempt including attempts after a process restart. A key held only in a local variable inside a retry loop is lost the moment the app is killed mid-payment, which is precisely when the retry matters most.

Where the key is generated on the client

Inside the retry loop

async function pay(amount) {
  for (let attempt = 0; attempt < 3; attempt++) {
    try {
      return await post('/payments', { amount }, {
        'Idempotency-Key': crypto.randomUUID(),   // new key every attempt
      })
    } catch (e) { if (!retryable(e)) throw e }
  }
}

With the intent, persisted

async function pay(amount) {
  // One intent -> one key, created when the user decided, and durable
  // so it survives an app restart mid-payment.
  const intent = await localStore.upsertPendingPayment({
    key: crypto.randomUUID(), amount,
  })

  for (let attempt = 0; attempt < 5; attempt++) {
    try {
      const res = await post('/payments', { amount },
        { 'Idempotency-Key': intent.key })
      await localStore.clearPending(intent.id)
      return res
    } catch (e) {
      if (!retryable(e)) throw e
      await sleep(backoffWithJitter(attempt))
    }
  }
}

On the left, every attempt is a distinct intent as far as the server can tell, so a server with a flawless implementation still charges three times. On the right the key is created once, outlives the process, and every attempt — including one made after a crash — presents the same identity.

What the server does with a key it has seen before

Receiving a known key is not one situation but four, and each needs a different response. The original may have completed successfully; it may have completed with a business failure; it may still be running; or it may have been presented with a different body, which means the client is confused.

The first two are replays and should return the stored response verbatim, including the original status code. A client that receives 201 the first time and 200 the second has to handle two shapes for one outcome, and will get it wrong.

The third is the concurrency case and is the hardest — it is covered in full in The Idempotency Key Flow, because the answer is a claim rather than a check. The fourth is a client bug, and the correct behaviour is to refuse loudly rather than to guess which body was intended.

Four meanings of "I have seen this key"

Trigger	Symptom	Cause	Response
Key exists, original succeeded	Client retried after losing the response	Response lost, not request	Return the stored status and body, unchanged. Do not re-execute.
Key exists, original failed a business rule	Client retries a request that will never succeed	Deterministic rejection — insufficient funds, invalid card	Return the stored error. Retrying cannot change the outcome.
Key exists, original failed transiently	Retry blocked by a stored 503	Stored a response that should not have been stored	Do not persist transient failures as final; release the key so a retry can proceed (Idempotency Storage).
Key exists, still in progress	Two concurrent executions of one intent	A check-then-act with no atomic claim	Atomic claim; the loser gets 409 with `Retry-After` (The Idempotency Key Flow).
Key exists, different request body	Wrong response returned for a real new intent	Client reused a key across intents	Reject with 422 and name the mismatched fields. Never guess.
Key exists but belongs to another principal	Cross-tenant response disclosure	Key store not scoped to the caller	Scope the lookup by principal; treat a foreign key as absent (Multi-Tenancy).

The key you send is as important as the key you receive

PROTOCOL-SPECIFICProviders differ in key length limits, permitted characters, retention window, and what they do when a key is reused with a different payload — some return the original response, some return an error. Truncating a hash to fit one provider's limit is safe; assuming another provider's semantics is not.

Your handler makes calls too, and your retries duplicate for exactly the same reason your clients' do. The payment provider is protecting itself with the same mechanism, and it is your job to use it.

The clean pattern is to derive the downstream key deterministically from the inbound one, so that any retry — the client's retry of your endpoint, your job runner's retry of the task, your HTTP client's retry of the call — produces the same downstream key and therefore the same single charge.

Determinism is what makes it work. A random key generated at call time is regenerated on retry and buys nothing. A key derived from stable inputs is reproducible from any attempt at any layer.

Deriving a downstream key that survives every retry layer

1import { createHash } from 'node:crypto'
2 
3// Deterministic: the same inbound intent always produces the same
4// downstream key, no matter which layer is retrying.
5function downstreamKey(inboundKey: string, purpose: string): string {
6  return createHash('sha256')
7    .update(`${inboundKey}:${purpose}`)
8    .digest('hex')
9    .slice(0, 64)                      // most providers cap key length
10}
11 
12async function chargeCard(intent: PaymentIntent, inboundKey: string) {
13  return psp.charges.create(
14    { amount: intent.amount, currency: intent.currency, source: intent.source },
15    { idempotencyKey: downstreamKey(inboundKey, 'charge') },
16  )
17}
18 
19async function refund(intent: PaymentIntent, inboundKey: string) {
20  // A different purpose -> a different key, so a refund is never
21  // deduplicated against the charge that preceded it.
22  return psp.refunds.create(
23    { charge: intent.chargeId },
24    { idempotencyKey: downstreamKey(inboundKey, 'refund') },
25  )
26}

The purpose segment is what stops two different downstream operations sharing a key. Without it, one inbound request that both charges and refunds would send the same key twice, and the provider would correctly return the first result for the second call.

How to build it

Most important first.

Require the key on every state-changing endpoint that a retry could duplicate. Optional idempotency is idempotency that is absent when it matters.
Generate it at the point the user expresses intent — when the button is pressed — not at the point the HTTP call is made, or a retry loop will generate a fresh one per attempt.
Store the request fingerprint with the key and reject a reused key carrying a different body with 422, naming the mismatch. A client reusing a key for a different intent has a bug you should not paper over (The Idempotency Key Flow).
Bound the key: maximum length, a permitted character set, and rejection of empty or whitespace keys. It is an attacker-controlled string that becomes a database key.
Document the retention window explicitly — "we honour a key for 24 hours" — because it is the client's only basis for deciding how long a retry remains safe.
Pass keys through to the external providers you call, deriving a deterministic key from your own so that your retry of the downstream call is also deduplicated (Idempotency in Backends).

What can go wrong

Failure modes

The client generating a new key inside its retry loop, so every retry looks like a new intent. The server is correct, the mechanism is defeated, and it can only be seen from the client side.
A key reused across genuinely different requests — often because it was derived from a stable domain id like order-42 rather than from the attempt — so a later legitimate operation is silently returned an old response.
A key scoped globally rather than per-caller, allowing cross-tenant collisions and, worse, cross-tenant response disclosure (Idempotency Storage).
Key retention shorter than the client's retry horizon: a client with a 24-hour retry queue against a 1-hour server window duplicates on late retries.
Requiring the key but treating a missing one as "generate one server-side", which produces a per-request key and silently disables the whole mechanism.
The key present but not propagated to the outbound provider call, so the API layer is idempotent and the charge is not.

What can race

Two retries carrying the same key arriving concurrently — the key alone does not resolve this; only an atomic claim on it does (The Idempotency Key Flow).
A client regenerating its key concurrently with a server-side replay, producing an intent the server cannot recognise as a repeat.
Key expiry running while a late retry is in flight, so the retry is treated as a new intent (Idempotency Storage).

Security

Keys must be namespaced by the authenticated principal. A globally-scoped key store lets one caller present another's key and receive their stored response (Object-Level Authorization).
Do not derive keys from predictable values on the server. If a key is guessable and the store is not scoped, replaying it retrieves someone else's result.
Bound length and count per caller. Unbounded attacker-supplied keys are an unbounded write channel into your storage (Resource Limits).
Do not log full stored responses against keys; the stored payload contains whatever the endpoint returns, which for payments is exactly the sensitive part (Secrets in Logs).
A key must not extend authorization in time. Re-check permissions on replay rather than assuming the original check still holds.

Misreads

"Hash the body" — this conflates identical requests with repeated requests. They are different things, and the difference is exactly what the key exists to express.
"The server should generate the key." The server cannot know that two requests are one intent; by the time it could, both have arrived.
"A request id and an idempotency key are the same." A request id identifies one HTTP attempt for tracing and differs per retry. An idempotency key identifies the intent and is stable across retries. Using one as the other breaks whichever purpose it was not built for (Correlation Ids That Survive Every Hop).
"Use the order id as the key." Then a legitimate second operation on that order — a partial refund, a second charge — is swallowed as a duplicate.
"The key makes the endpoint idempotent." The key is an identifier. The idempotency is in what the server does with it (The Idempotency Key Flow).

Operating it

How you see it in production

Ratio of requests carrying a key to those without, per client. A client at zero is a client that will double-charge.
Distinct keys per intent: if a client's retry rate is high and its distinct-key rate is equally high, it is regenerating keys per attempt — the most common client-side bug, and visible only in this comparison.
Count of key-reuse-with-different-body rejections. A spike after a client release means their key derivation changed.
Age distribution of keys at replay time, which tells you whether your retention window is longer or shorter than actual client behaviour.

What changes at 10x and 100x

Key volume equals request volume on protected endpoints — it grows exactly as fast as your traffic, not as fast as your duplicates.
At 10x, the index on (principal, key) becomes one of the busiest in the database; at 100x it usually wants its own store with native expiry (Idempotency Storage).
Retention dominates: window length multiplied by request rate is the steady-state row count, and that product is the only number that matters for capacity.

What this costs

Requiring a key makes the API harder to call for the simplest clients — a curl example now needs a UUID. Most public payment APIs accept that cost, and it is the reason some make the header optional, which weakens the guarantee.
A fingerprint check catches client bugs and means a legitimate change in a retried request — a client that adds a field on retry — becomes a 422 the client must handle.
Long retention makes late retries safe and grows storage linearly. Short retention keeps storage small and quietly reopens the duplicate window.
Deriving downstream keys from your own couples your key format to a provider's constraints on length and character set.

Where this applies

Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.

GENERALThe properties — client-generated, stable per intent, unique across intents, opaque to the server — hold for any API.
PROTOCOL-SPECIFICThe Idempotency-Key header name is a convention formalised in an IETF draft and adopted by several payment APIs; others use X-Idempotency-Key, a body field, or a provider-specific name, and their retention windows and mismatch behaviour differ. Read the specific provider's contract rather than assuming this one.
SCALE-SPECIFICFor an internal service with one client team, an agreed natural key is often enough and the header is ceremony. For a public API with third-party integrations you cannot patch, the explicit header is the only mechanism that survives clients you have never met.

Where the depth lives

This domain teaches the application-side mechanism and hands the rest off.

API Designidempotency-keys idempotency idempotency-vs-deduplication

Architectureidempotency

Securityreplay-attacks

Concurrencyreasoning-about-races shared-mutable-state

Domains that do not exist yet

— Distributed Systems — client-assigned identifiers as the general solution to deduplication across an unreliable channel.