FailureGENERALSCALE-SPECIFICILLUSTRATIVE

Duplicate Requests

A payment must not happen twice, and the double click, the browser retry, the webhook redelivery and your own retry loop all make the same request twice. Idempotency is the name for the answer; the lesson is discovering the question.

The moveWorked exampleNext questions

The situation, the reflex, and why it stalls

Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.

The question

The same checkout request arrives twice. What should happen, where is it decided, and how do you know it works?

The situation

Test-mode payments work. During a demo I clicked "Pay" twice because the button did not disable fast enough, and the provider dashboard showed two charges for one cart. I disabled the button. Then I wondered what happens when the network retries for me, and realised the button was never the problem.

The reflex

Disable the button after the first click. It is the visible cause, the fix takes a minute, and the demo no longer produces two charges. A spinner on the button looks like the problem being handled.

Why it stalls

The button was one of several ways the request repeats. Browsers retry some requests on their own; mobile networks drop responses and the app resubmits; the provider redelivers its notification; your own timeout-and-retry loop, added for resilience, resubmits the charge. Disabling the button fixed the only duplicate you had seen.

What the reflex produces — and fails to produce
  • The button was one of several ways the request repeats. Browsers retry some requests on their own; mobile networks drop responses and the app resubmits; the provider redelivers its notification; your own timeout-and-retry loop, added for resilience, resubmits the charge. Disabling the button fixed the only duplicate you had seen.
  • The fix lives in the layer least able to enforce it. The client can be a script, an old tab, or a second device, and none of those saw the disabled button. The invariant "one payment per checkout" is a property of the backend and the data, and the button cannot hold it.
  • Nothing distinguishes the second request from a new one, so even a backend that wanted to refuse it could not. The stall is not the missing check; it is the missing identity.
ProblemUnderstandRequirementsConstraintsUnknownsDecompositionSmallest StepModelExperimentObserveDebugLearnIterate

The move

Precisely enough to apply it to a problem you have never seen — not a slogan.

  • Name the invariant first: for each checkout attempt, at most one charge and at most one order. Write it as a sentence with "at most once" in it, because that phrase is what turns "the button double-fired" into a property of the system (Invariants in an Online Store, What Must Never Break).
  • Enumerate the ways a request repeats, from every layer that can send it: the human, the browser, the network, your retry code, the provider's redelivery, a background job that runs twice. The list is short and it is the same for most systems; what changes is which of them can reach the operation that must not repeat.
  • Give the operation an identity that is the same on every repeat and different on every new attempt, generated before the first send. For checkout that is a key created when the cart is confirmed, not when the request is built; a key that is generated per request is a new attempt every time and protects nothing.
  • Decide where the check lives and what a repeat returns. The check lives where the invariant can be enforced against concurrent repeats — usually the database, through a uniqueness constraint on the key — and the repeat returns the first attempt's result, so that the client cannot tell it was a repeat. The engineering of keys, storage and expiry is the Backend domain's; the move here is to arrive there with the invariant and the enumeration in hand (Invariants Under Concurrency).

The operation, and the identity that makes a repeat recognisable

The pseudocode shows the two halves of the move: a key created before the first attempt, and a reservation of that key that happens before anything with a lasting effect. The framework's idempotency middleware, when you get to it, is this shape with storage and expiry added.

What to notice: the reservation is an insert with a unique constraint, not a lookup followed by an insert. The difference is invisible in a sequential test and decisive when two repeats arrive together.

Checkout, at most once
1on review page shown:
2 attemptKey = new id # created once per checkout attempt, kept by the client
3
4function checkout(cart, attemptKey):
5 reserved = insert attemptKey into attempts # unique constraint on attemptKey
6 if not reserved:
7 return stored result for attemptKey # a repeat: same answer as the first time
8 order = create order (pending) for cart
9 charge = provider.charge(total, key = attemptKey)
10 record charge on order
11 store result for attemptKey = confirmation(order)
12 return that result

The key travels with every submit of the same attempt and changes only when the customer starts a new one. If the provider accepts a key, it is the same key — one identity for the whole attempt across both systems.

The why ladder from "we need a lock"

The first idea most people reach for is a lock: hold the cart while checkout runs so a second submit waits. The ladder shows what the lock was standing in for, and that the real requirement is satisfied by something with no waiting in it.

Why ladder

We need a distributed lock on the cart during checkout.

  1. Why a lock? So a second checkout submit cannot run while the first is running.
  2. Why must it not run? Because it would charge the card again and create a second order.
  3. Why would it do that? Because it cannot tell it is a repeat of an attempt already in progress or already finished.
  4. Why can it not tell? Because nothing identifies the attempt; each request looks new.
real requirement At most one charge and one order per checkout attempt, and a repeat receives the first attempt's result — including repeats that arrive after the first has finished, which a lock does nothing about.
simpler An attempt key generated by the client before the first submit and reserved with a unique constraint before the charge; a repeat finds the reservation and returns the stored result.

the claim was right when The operation cannot be given an identity — for instance an external system that must be called exactly once per second across many workers — or when the work must be serialised for a different reason, such as inventory allocation across many carts.

Where the check can live

The decision is not which mechanism is best but which layer can actually hold the invariant against every source on the repeat list. Each option stops some sources and misses others, and the honest design usually has two of them: a client-side guard for the human, and a data-level constraint for everything else.

Where to stop the repeat

Which layer enforces at-most-once for checkout?

Client: disable the button, block resubmit

when Always, as a courtesy — it stops the human and improves the experience.

cost Stops nothing that does not go through this client: a retry, a second tab, a script, a redelivery.

Application: check a key in memory before proceeding

when A single process, low stakes, or a prototype where the point is to learn the pattern.

cost Fails silently with a second instance; a read-then-write passes two concurrent repeats.

Data: unique constraint on the attempt key, reserved before the external call

when Any operation whose repeat costs money or data — the checkout charge, the order insert, the notification handler.

cost A key table to store, replay and expire; the client has to generate and keep the key.

Provider: pass the key to the provider's charge call

when The provider supports it — most payment providers do. Use it in addition to the data-level check, never instead of it.

cost Covers only the charge; the order, the stock and the email are still yours to protect.

How to do it

Most important first.

  • Write the at-most-once sentence for each operation with a lasting external effect: charge, send email, decrement stock, create shipment. Not every operation needs one; reads never do.
  • For each, list who can repeat it. If the provider can redeliver a notification, the notification handler is on the list even though no human is involved.
  • Choose the key's lifetime: per cart confirmation, per order, per notification id. The key must be created before the first attempt and survive the client dying.
  • Put the uniqueness where concurrent repeats are serialised — the database — and test with two simultaneous requests, not two sequential ones (Example-Driven Thinking).
  • Decide the repeat's response: return the stored result, not an error. A client that gets an error on a repeat will retry again.

Worked on a concrete problem

The move has to produce something. This is what it produced.

  • The store's at-most-once operations: charge a card (per checkout attempt), create an order (per checkout attempt), decrement stock (per order line), send confirmation (per order), mark paid (per provider notification). Reads of the catalog and the cart are not on the list.
  • Who repeats checkout: the customer double-clicking; the browser resubmitting a form on refresh; the mobile app retrying a lost response; our own retry on provider timeout; a support agent re-running a failed job. Who repeats "mark paid": the provider redelivering, and our own job re-processing a batch after a crash.
  • The key for checkout is created when the customer reaches the review page and sent with every submit of that review. The backend inserts the key into a table with a unique constraint before calling the provider; a second insert fails, and the handler returns whatever the first attempt stored. Two simultaneous submits in a test produce one row, one charge and two identical responses.
  • The "mark paid" handler stores the provider's notification id with the same pattern, so the redelivered notification is recognised and the second stock decrement and email never happen.

How you know it worked

What now exists that did not before, and what question you can now ask.

  • Each operation that must not repeat has a written at-most-once sentence, a key with a stated lifetime, and a place where the uniqueness is enforced.
  • A test exists that submits the same checkout twice at the same time and observes one charge, one order and two matching responses.
  • The word "idempotent" now names something specific in your system rather than a property you hope your endpoints have.

The questions you can now ask

The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.

Next questions
  • ?Which operations in this system must happen at most once, and what is the sentence that says so?
  • ?Who can send this request twice — human, browser, network, my retry, their redelivery, a job?
  • ?What identity is the same on a repeat and different on a new attempt, and when is it created?
  • ?Where is the uniqueness enforced against two repeats arriving at the same instant, and what does the second one receive?

What can go wrong

How the move itself fails
  • Every endpoint gets an idempotency key, including reads and operations that are naturally safe to repeat, and the key table becomes the busiest table in the database. The enumeration decides which operations need it; most do not.
  • The key is generated on the server per request, so every request is unique and the mechanism exists without protecting anything. The key's lifetime is the design; the table is plumbing.
  • The check is done in application code with a read-then-write, and passes every sequential test while two concurrent repeats both pass the read. Concurrency is the case; test it.
What the move costs
  • A key table is state that has to be stored, indexed and eventually expired; the stored response has to be large enough to replay and small enough to keep.
  • Returning the first result to a repeat hides from the client that anything was repeated; sometimes the client wanted to know, and the design has to decide.
Misreads
  • "Idempotency means the request has no effect the second time." It means the second time has the same effect as the first — one charge, one order, the same response — which is stronger and more useful than "no effect".
  • "A unique constraint on the order table is enough." It stops two orders; it does not stop two charges if the charge happens before the insert. The key has to be reserved before the external call, which is the ordering question from Partial Failure.
  • "The provider handles this for me." Many providers accept an idempotency key on the charge call — and it is your key, with your lifetime, and their guarantee covers only their side. The notification handler and the order write are still yours.

Where this applies

Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

  • GENERALThe enumeration of who repeats a request holds for any system with a client and a network; the operations that need at-most-once are domain-specific — in a chat app it is "send message" and in a URL shortener it is nothing at all.
  • SCALE-SPECIFICOn one server with one process, an in-memory check and a database constraint are both fine; with several instances only the database — or something else shared and serialised — can hold the invariant, and the in-memory version silently stops working when the second instance starts.
  • ILLUSTRATIVEThe demo, the double click and the two charges are invented; the list of repeat sources is the shape of the argument and not a complete catalogue for any real provider.

Where the depth lives

This domain asks the question and hands the answer off by name.