Invariants Under Concurrency
Product A has three units. Alice buys two; Bob buys two at the same moment. Every line of both checkouts is correct and the store has promised four units of three. The example reveals the concurrency problem that no single-user test can see, and the thinking move is to name it as a question before choosing a mechanism.
The situation, the reflex, and why it stalls
Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.
An invariant holds for every action taken alone. How do you find out whether it survives two of them at once — and what do you decide once you know it does not?
Checkout works. I tested it thoroughly: buy, refund, sold out, all fine. Then someone asks what happens if two people buy the last unit at exactly the same time, and I genuinely do not know. I think the database handles that? I am not sure what "handles" would even mean.
Wrap the checkout in a transaction. Transactions are what the database offers for exactly this, the word sounds like "all or nothing", and adding one is a single line. It feels like the concurrency problem has been delegated to something that understands it better than you do.
A transaction makes a sequence atomic with respect to failure, not necessarily with respect to other transactions. At the default isolation level of most databases, two transactions can both read stock 3, both decide there is enough, and both commit a decrement. The line was added, the test still passes, and the invariant still breaks — now with a transaction around it.
- A transaction makes a sequence atomic with respect to failure, not necessarily with respect to other transactions. At the default isolation level of most databases, two transactions can both read stock 3, both decide there is enough, and both commit a decrement. The line was added, the test still passes, and the invariant still breaks — now with a transaction around it.
- The problem was never named. "Does the database handle it?" is not a question that can be researched, so it is answered with a feeling, and the feeling is that transactions are the answer. The actual question — which interleavings of read and write are possible, and which of them violate the invariant — was never written down.
- Because nothing was named, nothing can be tested. A single-user test cannot produce the interleaving; without a stated interleaving to reproduce, the "fix" is unverifiable and stays in the code as a hope.
The move
Precisely enough to apply it to a problem you have never seen — not a slogan.
- Take the invariant and the actions that touch its state, and construct the interleaving by hand: two actors, each performing the same action, with the individual steps — read, decide, write — listed and then shuffled. The example from Finding Invariants From Examples is the tool; the difference here is that you enumerate the interleavings deliberately rather than waiting for one to feel wrong.
- Name the exact problem the interleaving reveals, in words that could be researched: "two transactions read the same stock value before either writes; the second write is based on stale data". That sentence is a known shape — a lost update, a check-then-act race — and naming it turns "does the database handle it?" into a question with an answer in the Database and Concurrency domains (Concurrency Anomalies, Reasoning About Races: A Method, Not an Instinct).
- Only then choose a mechanism, and choose it by what it guarantees against the interleaving you wrote down: an atomic conditional update (
decrement where stock >= n) removes the gap between check and act; a row lock serialises the two checkouts; an optimistic version column makes the second write fail; a serializable isolation level makes the database detect the conflict. Each has a cost and a failure mode, and the choice is a trade-off, not a fact (Trade-Off Thinking). - Finally, ask what the business wants when the race is lost. Refuse Bob? Backorder him? Oversell and apologise? The mechanism can only enforce a decision that has been made; "stock never negative" was an invariant, and "what Bob sees" is a requirement that the race forced into the open (Requirements Emerge During Implementation).
The interleaving, written out
The store example, as steps. Each actor's steps are individually correct and the invariant survives either actor alone. The shuffled ordering is the one that matters; writing it out is what turns "I think the database handles it" into a thing that can be reproduced and researched.
The last block shows the mechanism that closes this particular gap by making the check and the write one operation. It is not the only mechanism, and the section after this one is about choosing.
1checkout(actor, qty):2 s = read stock(A) # step 13 if s >= qty: # step 2 <- the decision uses s4 write stock(A) = s - qty # step 3 <- the write is based on stale s5 create order(actor, qty) # step 46 7interleaving that breaks "stock never negative":8 Alice 1 (s=3) Bob 1 (s=3) Alice 2 ok Bob 2 ok9 Alice 3 (stock=1) Bob 3 (stock=1) -> two orders, 4 units promised of 310 with "stock = stock - qty" instead: 3 -> 1 -> -111 12closing the gap by removing it:13 rows = update inventory set stock = stock - qty14 where product = A and stock >= qty15 if rows == 0: refuse(actor, "sold out") # Bob lands here16 else: create order(actor, qty)Notice the fix is not "add a transaction". It is "make the decision and the write one indivisible operation", which a conditional update does and a plain transaction at default isolation does not.
Choosing the mechanism by the guarantee, not by habit
Each option below closes the gap in a different way and opens a different one. The criteria are the guarantee against the interleaving you drew, what it costs on the happy path, and the failure mode it introduces. The store's decision depended on a product question — whether stock is reserved while the customer pays — as much as on any of these.
Two actors read, decide and write on the same state. Which mechanism prevents the interleaving that violates the invariant?
when The check and the write are on the same row and can be one statement. Simplest, fastest, no lock held across slow work.
cost Cannot span rows or tables, cannot hold a reservation while something slow happens.
when Several rows or several steps must be decided together; the critical section is short.
cost The lock is held for the whole section — a payment call inside it stalls every other buyer; deadlocks become possible (Locks and Deadlocks).
when Conflicts are rare and the write can be retried; long-running edits (admin editing a product) where a lock would be held for minutes.
cost The loser retries or fails; under contention, retry storms (Optimistic Concurrency: Versions and If-Match).
when The invariant spans reads and writes that are hard to express as one statement, and the database can detect the conflict for you.
cost Transactions abort on conflict and must be retried everywhere; throughput drops; not every database implements it the same way.
when Violation is rare, cheap to detect with a reconciliation query, and cheap to repair — not stock or money.
cost The invariant is allowed to break; only acceptable when the business has said so.
"We need a distributed lock"
The race sometimes gets a bigger answer than it needs. The ladder below is a claim heard on real teams once the interleaving has scared them; walking it down finds what was actually required and the simpler thing that meets it — and the case where the claim was right.
“We need a distributed lock service so that two checkouts cannot decrement stock at the same time.”
- ↓Why a lock? So that only one checkout at a time can read and decrement the stock row.
- ↓Why must the read and decrement be exclusive? Because the decrement is based on the value read, and a stale read produces a wrong write.
- ↓Why must the write be based on the read? It does not have to be: the database can decrement conditionally in one statement, and the read disappears.
- ↓Why distributed? Because there are several application servers — but they all talk to one database, which already serialises writes to a row.
the claim was right when The stock lives in a store the database cannot coordinate — a cache, a separate inventory service, several databases sharded by warehouse — or the critical section spans systems, such as reserving stock here and a delivery slot in another service. Then a coordination mechanism outside any one database is the honest answer, with its own failure modes (A Mutex on Server A Does Nothing About Server B).
How to do it
Most important first.
- Write the action as its atomic steps — read, decide, write — before thinking about the database. If the decision depends on the read and the write comes after, there is a gap, and the gap is where the race lives.
- Interleave two copies of the steps by hand and find the ordering that violates the invariant. Write that ordering down; it is the reproduction and the test.
- Name the shape: lost update, check-then-act, double submit, phantom. The name is what makes it searchable and what makes the mechanism list findable.
- Pick the mechanism by the guarantee it gives against your ordering, and write down its failure mode — deadlock, retry storms, false conflicts — as the new thing to watch.
- Decide, with whoever owns the product, what the losing actor experiences. Then test the interleaving, not the happy path (Stress Testing: A Test That Passed Once Proves Nothing).
Worked on a concrete problem
The move has to produce something. This is what it produced.
- Product A, stock 3. Checkout as steps: (1) read stock, (2) if stock ≥ 2 continue, (3) write stock − 2, (4) create order. Interleave Alice and Bob: A1 B1 A2 B2 A3 B3 A4 B4. Both read 3, both pass the check, both write 1 — or, with
stock = stock − 2, stock ends at minus one — and two orders exist. Named: a check-then-act race with a lost update; the gap is between step 2 and step 3. - Mechanism by guarantee. A single statement
UPDATE inventory SET stock = stock − 2 WHERE product = A AND stock >= 2removes the gap: the check and the write are one atomic operation, and the second statement affects zero rows, which the code treats as "sold out". Cost: nothing extra; failure mode: none for this invariant, but it does not reserve stock for the customer who is still typing a card number. ASELECT … FOR UPDATEbefore the check serialises the two checkouts instead (Pessimistic Locking) — simpler to reason about, and now a slow payment step holds a lock. Serializable isolation would catch the conflict and abort Bob's transaction, at the cost of retries everywhere (Isolation Levels). - The decision the race forced: the store had never said when stock is committed. If it is decremented at checkout, Bob is refused immediately but Alice's failed payment has to restock. If it is decremented on payment confirmation, both may reach the provider and one is refused after entering a card. The founder chose "reserve at checkout, release after a timeout" — a requirement that did not exist until the interleaving was drawn, and a new invariant: "reserved plus available never exceeds existing".
How you know it worked
What now exists that did not before, and what question you can now ask.
- A written interleaving exists that violates the invariant, using only correct individual steps, and it has a name.
- The chosen mechanism is justified by the guarantee it gives against that interleaving, and its own failure mode is written next to it.
- A test reproduces the interleaving — two concurrent checkouts for the last units — and asserts the invariant, and it fails without the mechanism.
- A product decision that did not exist before — what the losing customer sees, when stock is committed — is now written down.
The questions you can now ask
The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.
- ?What are the atomic steps of this action, and is there a gap between a read that decides and a write that acts?
- ?Which interleaving of two copies of these steps violates the invariant, and what is that shape called?
- ?Which mechanism closes that specific gap, what does it guarantee, and what is its own failure mode?
- ?When the race is lost, what does the losing actor experience — and who has decided that?
- ?Does the interleaving reappear if these steps ever live in two services or two databases?
What can go wrong
- Every write in the system is examined for races and the design drowns in locks. The move applies to invariants whose violation matters — money, stock, uniqueness — and not to a product description that two admins might both edit.
- The mechanism is chosen before the interleaving is drawn — "we use optimistic locking here" — and it does not cover the actual ordering. Serializing the decrement does nothing for a double payment confirmation; each invariant has its own interleaving.
- The race is fixed within one database and reintroduced across two: the stock check happens in the order service, the decrement in the inventory service, and there is no transaction spanning them. The interleaving reappears one level up (Partial Failure).
- The concurrency problem is solved and the product decision is skipped: Bob gets a stack trace because nobody decided what "sold out at the last moment" looks like.
- Drawing interleavings takes longer than adding a transaction, and for state that nobody will ever race on it finds nothing.
- Every mechanism costs something: locks hold during slow steps, optimistic versions force retries, serializable isolation aborts transactions that did not really conflict.
- Naming the race forces a product decision earlier than the product team wanted, and the decision — refuse, backorder, oversell — is not the engineer's to make alone.
- "Transactions solve concurrency." They solve atomicity with respect to failure and, at some isolation levels, some interleavings. Whether yours is one of them depends on the level and the statements; the interleaving on the page is how you find out (Transactions and ACID).
- "This only matters at scale." Two customers buying the last unit during a launch is a scale of two. Frequency rises with load; possibility does not depend on it, and the double charge arrives at whatever scale the store has.
- "The single-statement update is always the answer." It is the answer for check-and-decrement. It cannot reserve stock while the customer pays, cannot span two tables, and says nothing about the double confirmation. Each invariant gets its own interleaving and its own mechanism.
Where this applies
Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALAny invariant over state that more than one actor can change — a counter, a unique username, a seat on a flight — is subject to an interleaving, and the move of listing steps and shuffling them is the same in a database, a process or a browser tab.
- SCALE-SPECIFICThe possibility of the race does not depend on scale; its frequency does. At a handful of orders a day a store may reasonably decide to detect and repair rather than prevent; once the last-unit race happens weekly the mechanism is no longer optional.
- ILLUSTRATIVEThree units, two buyers of two, and the reserve-then-release decision are invented so that the interleaving fits on one screen; the step numbering is a teaching device, not a description of any database's execution.
Where the depth lives
This domain asks the question and hands the answer off by name.