Idempotency

A client cannot distinguish "my request was lost" from "my request succeeded and the response was lost". From the outside both look like a timeout. The only safe client behaviour is to retry, and the only safe server behaviour is to make retries harmless. The order service that charges a card in 200 ms usually, then once hits a 30 s gateway timeout because the payment provider was slow — and charges again when the client retries — is not a rare bug; it is the default behaviour of every non-idempotent write on a network.

The same problem appears wherever there is a retry: a message queue redelivers after a consumer crash (Message Queues), a worker re-runs a job after a lease expires (Background Jobs and Workers), a saga step is re-executed after an orchestrator restart (Saga Pattern), an agent retries a tool call after a timeout (Tool Errors, Retries and Timeouts). "Exactly once" is not a delivery guarantee any network can give; it is a property you construct on the receiving side, from at-least-once delivery plus an idempotent receiver.

The mechanism used by every payment API: the client generates a key (a UUID) for each logical operation and sends it as Idempotency-Key. The server stores the key with the response it produced. A second request with the same key does not re-execute anything — it returns the stored response, status code and all. The key must be chosen by the client because only the client knows that two HTTP requests are the same *intent*; a server-generated id would be different for each retry.

Details that matter in production. Keys are scoped: per user or per API credential, so two customers who both happen to send key 1 do not collide, and usually per operation type. Keys have a TTL — 24 h is common — because storing every response forever is a database of its own. The store must reject a *different* payload under the same key (422), otherwise a bug that reuses keys silently returns the wrong customer's response. And while the first request is still in flight, a duplicate must wait or return 409, not start a second execution: the check-then-insert must be atomic. Redis SET key value NX EX 86400 does exactly that in one command; a unique index on (user_id, key) does it in SQL, and the insert failing is the signal that this is a retry.

Some operations are idempotent by shape and need no key. PUT /users/42 { name: "Ada" } sets a value; doing it twice sets the same value. DELETE /orders/7 twice leaves the order equally deleted (return 204 both times, not 404 on the second). An UPSERT ... ON CONFLICT DO UPDATE keyed on a natural id is idempotent; INSERT is not. SET balance = 80 is idempotent; UPDATE balance = balance - 20 is not — and the difference between those two statements is most of the difference between a system that survives retries and one that does not. Designing the API so writes are *replacements keyed by identity* rather than *increments* removes the need for a key store for most endpoints; keep keys for the operations that are inherently one-shot (charge, send, create).

Systems that advertise "exactly-once" — Kafka transactions, stream processors — implement it as at-least-once delivery plus deduplication by sequence number inside a bounded scope, and the guarantee stops at the boundary of that scope. The moment the effect leaves the system (an email, a card charge, an HTTP call to someone else) it is at-least-once again, and the receiver needs its own idempotency. That is why the honest design statement is: deliver at least once, make every receiver idempotent. The interview question exactly-once is testing whether you know this.

A queue consumer or event handler gets the same message twice whenever it crashes between doing the work and acknowledging. The standard defence is a processed-message table: inside the *same* database transaction as the business write, insert (consumer_name, message_id) into a table with a unique constraint. If the insert conflicts, the message was already handled — roll back and acknowledge. Because the check and the effect commit atomically, a crash at any point leaves either both or neither. This only works when the effect is in the same database; when the effect is an external call, fall back to keys the external system honours.

Ordering interacts with idempotency: an OrderUpdated that arrives twice is harmless if applied as "set status to shipped", harmful if applied as "advance status by one". Design event handlers as upserts of a target state where possible, and carry a version or sequence number so a stale duplicate (v3 arriving after v4) is ignored rather than applied — the same discipline Event-Driven Architecture and Kafka-Style Logs: Topics, Partitions, Offsets rely on. The lock-free version of the processed-message check is a race under the default isolation level; see Isolation Levels for why a unique index, not a SELECT first, is the correct guard.