The question this answers
My code is correctly locked and it still double-processed. What did the lock actually protect?
A webhook receiver that checks "have we already processed this delivery id?", guarded by a process-local mutex, running behind a load balancer on four instances.
What the developer believed was shared: a processed set and a mutex. What is actually shared across instances: nothing in memory — only the database and the third-party endpoint they both call.
Each webhook delivery id is processed exactly once across the entire fleet, not once per instance.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
The bug, in the simplest possible form
A mutex is an agreement between threads in one address space. It has no representation outside that address space, no name any other process can refer to, and no way to make another machine wait. Four instances behind a load balancer have four mutexes, four processed sets, and four independent, perfectly-correct locking implementations that between them provide no exclusion whatsoever.
This is not a subtle bug in the usual sense — once stated it is obvious. What makes it the most common one is that *it is invisible until deployment topology changes*. In development there is one process, so the local lock and the global invariant coincide. Every test passes. The first production incident arrives with the second instance, and it arrives as duplicate side effects rather than as an error, so nothing alerts.
It generalizes past locks. A rate-limit counter in a variable limits each instance, so four instances allow four times the intended rate. An in-process cache with "only one refresh at a time" logic performs four refreshes. A once flag runs the initialization four times. Anywhere a per-process object stands in for a fleet-wide fact, the same failure exists.
| # | Load balancer | Instance A (mutex A) | Instance B (mutex B) | Payment provider | State |
|---|---|---|---|---|---|
| 1 | webhook delivery d-991 arrives; provider retries after a slow ack | · | · | · | deliveries in flight=2 |
| 2 | routes copy 1 to Instance A, copy 2 to Instance B | · | · | · | A processing=d-991 B processing=d-991 |
| 3 | · | lock(mutexA) | · | · | mutexA=held by A mutexB=free |
| 4 | · | · | lock(mutexB) — grants immediately; different process, different lock | · | mutexA=held by A mutexB=held by B |
| 5 | · | processedA.has("d-991") -> false | · | · | processedA=empty processedB=empty |
| 6 | · | · | processedB.has("d-991") -> false | · | processedA=empty processedB=empty |
| 7 | · | POST /refund 49.00 (no idempotency key) | · | · | refunds=1 |
| 8 | · | · | POST /refund 49.00 | · | refunds=2 ✕ Each delivery id is processed exactly once. Two refunds were issued for one delivery, and both instances' locks behaved perfectly. |
| 9 | · | · | · | both refunds succeed; customer is refunded twice | refunds=2 customer balance=+98.00 |
| 10 | · | processedA.add("d-991"); unlock(mutexA) | · | · | processedA=d-991 processedB=d-991 |
The three fixes, and the failure mode of each
The first and best fix is a database constraint or conditional write. INSERT INTO processed_deliveries (id) VALUES ('d-991') with a primary key on id makes the second instance fail with a duplicate-key error, which it treats as "already handled". The exclusion is enforced by the one component both instances genuinely share. Its failure mode is that it only covers what the constraint expresses, and that the work must be ordered so the claim happens *before* the side effect — claiming after the refund protects nothing.
The second is a lease from a coordination service: acquire a named lease for d-991, do the work, release. Its failure mode is everything in What Changes When the Shared State Is on Another Machine — a paused holder past its TTL, split-brain without fencing, and a new availability dependency on the critical path. It is the right tool when the work is long and cannot be made idempotent, and the wrong tool when a constraint would have done.
The third is idempotency at the destination: send an idempotency key with the refund, and let the payment provider deduplicate. Its failure mode is that it depends on someone else's implementation and deduplication window, and it does not help for side effects that have no such mechanism — an email, a physical shipment, a webhook you emit. See Idempotency Keys: The Mechanism and Consumer-Side Idempotency.
A fourth non-fix deserves naming because it is attempted constantly: sticky sessions, so the same delivery id always lands on the same instance. It reduces the frequency and does not remove the failure, because a deploy, a scale event or a health-check eviction moves the assignment while work is in flight. Reducing a race's probability without changing its possibility is not a fix.
How to spot it before production does
The review question is mechanical and worth making a habit: *for every lock, counter, cache and flag — what is the scope of the invariant it protects, and what is the scope of the thing protecting it?* If the invariant says "across the system" and the mechanism lives in one process's heap, you have found this bug regardless of how correct the local code is.
A second heuristic: any variable whose name contains "already", "once", "seen", "processed", "in flight" or "pending" is a candidate. Those words describe fleet-wide facts almost every time they are written, and they are almost always implemented as process-local state.
The third is a deployment-topology check. Ask what happens when the instance count goes from one to two. If the answer changes any user-visible behaviour — rate limits, deduplication, scheduled jobs, cache refresh, warm-up initialization — that behaviour is process-scoped and is either a bug or a documented approximation. Both are fine; assuming it is neither is not.
| Mechanism | Believed invariant | Actual behaviour at N instances | Correct mechanism |
|---|---|---|---|
mutex around a handler | One handler runs at a time | N handlers run at a time, one per instance | Database constraint, or a lease with fencing |
In-memory processed set | Each id handled once | Each id handled up to N times | Unique row claimed before the side effect |
| In-process rate-limit counter | X requests per second allowed | N x X requests per second allowed | Shared counter store, or a limit at the gateway |
initialized boolean | Initialization runs once | Runs N times, concurrently | Idempotent initialization, or a lease |
| Single-flight / request coalescing | One refresh per key at a time | N refreshes per key at a time | Shared lock or a shared cache with an atomic claim |
| Cron guarded by a local flag | The job runs once per schedule | Runs N times per schedule | A leader lease, or a uniqueness claim on the run key |
| Sticky sessions | "Same key always hits the same instance" | Until a deploy, scale event or eviction moves it mid-flight | Not a fix — reduces probability, not possibility |
Key points
- A mutex is an agreement between threads in one address space. It has no meaning to another process and cannot make another machine wait.
- N instances behind a load balancer have N locks, N caches and N counters, each perfectly correct and collectively providing no exclusion.
- It is invisible in development because one instance makes the local scope and the global scope coincide, so every test passes.
- It generalizes past locks: rate-limit counters,
initializedflags, dedupe sets and single-flight coalescing all multiply by the instance count. - The fix is to enforce exclusion where the state is genuinely shared — a database constraint, a lease with fencing, or idempotency at the destination.
- Sticky sessions are not a fix. They reduce the probability of the race without removing its possibility, and deploys move assignments mid-flight.
- The review question: what is the scope of the invariant, and what is the scope of the mechanism? A mismatch is this bug.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • A process-local lock is a data structure in one heap; acquiring it consults only that structure, so no other process is consulted or delayed.
- • A load balancer distributes deliveries across instances with no knowledge of which ids are in flight where, so two copies of the same work routinely land on different instances.
- • Each instance independently performs check-then-act against its own state, and each independently concludes the work has not been done.
- • The side effect lands on a genuinely shared resource — a payment provider, a database, an email gateway — which is the only place the duplication becomes observable.
- • A correct mechanism performs an atomic claim at that shared resource *before* the side effect, so the second claim fails and the second side effect never happens.
- • Delivery d-991 routed to A and B: both lock their own mutex, both find their own set empty, both refund. Two refunds, zero errors, both locks correct.
- • Rate limiter of 100/s per instance across four instances: the upstream receives 400/s and rate-limits the fleet, which presents as intermittent 429s that no single instance's metrics explain.
- • Nightly cron guarded by an
alreadyRanTodayflag: all four instances wake at 02:00, all four flags are false, four reports are generated and the last write wins. See What Changes When the Shared State Is on Another Machine. - • Single-flight cache refresh: on expiry, each of four instances coalesces its own concurrent requests into one refresh, so the origin receives four refreshes rather than one — a thundering herd divided by four rather than eliminated. See Thundering Herd and Single-Flight Coalescing.
- • With a unique constraint claimed first: A inserts
d-991and proceeds; B's insert is rejected with a duplicate key and it returns success without acting. One refund, and the duplicate is invisible to the customer.
- • A process-local mutex guarantees mutual exclusion among threads of that process. That guarantee is complete, correct, and irrelevant to a fleet-wide invariant.
- • A unique constraint guarantees at most one successful claim across every process and every future code path, which is the strongest guarantee available here and the cheapest.
- • A lease guarantees at most one grant, not at most one believer — enforcement still requires a fencing check at the resource.
- • Idempotency at the destination guarantees deduplication only within that destination's window and only for operations it supports; it says nothing about your other side effects.
- • Sticky routing guarantees nothing. It is an optimization that changes probabilities.
- • Moving exclusion to the database converts an in-memory lock into a row lock plus a network round trip, so the cost per operation rises by orders of magnitude.
- • A single claim row for a hot key serializes the entire fleet through it, which is correct and is now a fleet-wide contention point — What Contention Actually Costs.
- • A coordination service becomes a shared dependency whose latency and availability bound everything that claims through it.
- • Idempotency keys shift the contention to the destination, whose deduplication may itself have rate limits.
- • Duplicate side effects: double refunds, double emails, double shipments — with no error anywhere, because every component succeeded.
- • Rate limits multiplied by instance count, producing upstream 429s that no per-instance dashboard explains.
- • Scheduled jobs running once per instance, with last-write-wins deciding which result survives.
- • Cache stampedes divided by instance count rather than eliminated, so the origin still sees a burst.
- • The bug appearing only after a scale-up or a deploy, which makes the correlation with the code change invisible.
- • A claim written *after* the side effect, which records history correctly and prevents nothing.
- • Recognizing it in code review, which is where it is cheapest to fix and where the invariant-scope question reliably catches it.
- • During any scale-out from one instance to more, as an explicit checklist item rather than a discovery.
- • When diagnosing duplicate side effects with clean logs, since "every instance thought it was first" explains an otherwise impossible incident.
- • Never — but the fixes have costs, and reaching for a distributed lock when a unique constraint would do buys an availability dependency for nothing.
- • Over-correcting: adding fleet-wide coordination to operations that were always safe to repeat, paying round trips to prevent a harmless duplicate.
- • Treating a local lock as "good enough because it reduces duplicates", which is exactly the sticky-session error and leaves an unbounded failure in place.
- • Duplicate-side-effect counts measured at the destination — duplicate charges, duplicate emails, duplicate rows — not inferred from application logs.
- • Whether any behaviour changes when instance count changes; run the same load against one instance and four and compare rate-limit rejections and job execution counts.
- • Rejected duplicate-key claims, which is the healthy signal that the correct mechanism is working and how often it is needed.
- • Upstream 429 rate divided by instance count, which reveals a per-instance limiter masquerading as a fleet-wide one.
- • Grep for
Map,Set,once,initialized,processedandinFlightat module scope — a static-analysis-grade heuristic that finds most instances of this.
- • The correct fix moves a memory operation onto the network, adding latency, a failure path and a retry policy to something that used to be a function call.
- • The claim must be ordered before the side effect, which often means restructuring the handler rather than adding a line.
- • Failure handling grows: what happens if the claim succeeds and the side effect fails? Now you need compensation or a retryable claim with a TTL.
- • Local locks may still be needed *in addition*, for intra-process correctness, so both mechanisms coexist and their scopes must be documented.
- • Make the operation idempotent so duplicates are harmless — removes the need for exclusion entirely and is the most robust answer. See Idempotency Keys: The Mechanism.
- • A unique constraint or conditional write claimed before the side effect — cheapest real enforcement, and it never forgets.
- • Deterministic partitioning so only one instance is ever responsible for a given key, which removes contention rather than managing it.
- • A single-consumer queue per key, converting exclusion into ordering — Message Passing.
- • A lease from a coordination service, when the work is long and none of the above apply — with fencing, and with the failure modes from What Changes When the Shared State Is on Another Machine accepted explicitly.
What people believe, and what is true
The handler is properly locked, so it cannot run twice.
It cannot run twice *in that process*. With four instances it runs four times concurrently, and every lock behaves exactly as specified.
We use sticky sessions, so the same key always hits the same instance.
Until a deploy, an autoscale event, a health-check eviction or a rebalance moves it — usually while work is in flight. Sticky routing changes probability, not possibility.
It works in staging with one instance, so the logic is right.
One instance is precisely the configuration in which this bug is undetectable. Staging with one replica tests the local lock and never tests the invariant.