The question this answers
Given a function I did not write, how do I systematically find the schedule that breaks it?
A promo-code redemption handler: if (promo.remaining > 0) { promo.remaining -= 1; grantDiscount(user) }, run by every checkout request against a promo limited to 100 uses.
promo.remaining, an integer in a shared map of active promotions. grantDiscount writes to the user's record — a different location, and a second shared access most reviewers do not count.
The number of granted discounts never exceeds 100: grantsIssued + promo.remaining === 100 at every instant, and promo.remaining >= 0.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
The shape: check, gap, act
Almost every race condition a service engineer meets is one shape wearing different clothes. Something is observed, a decision is made from that observation, and the decision is acted on — with a gap in between during which the observation may stop being true. if (balance >= 100) withdraw(100). if (!exists(path)) create(path). if (!cache.has(k)) cache.set(k, expensive()). if (promo.remaining > 0) promo.remaining -= 1. The operating-systems literature calls it time-of-check to time-of-use; in application code it is usually just called "the bug".
The critical realisation is that the check produces a *fact about the past*. promo.remaining > 0 was true at the instant of the read and carries no promise about the instant of the write. Everything downstream of the check is acting on history. The bug is not that the check is wrong; it is that the code treats a historical fact as a current one.
Two things widen the gap dramatically and are worth spotting on sight. An await between the check and the act turns a nanosecond window into a network round trip. And a check performed in one service against a decision acted on in another turns it into a distributed problem where no local primitive helps at all. The version below has both problems in one function.
1async function redeem(promoId: string, userId: string) {2 const promo = promos.get(promoId) // (1) shared read3 if (promo.remaining <= 0) return { ok: false, reason: 'exhausted' }4 5 // <-- GAP 1: another task may decrement to 0 here6 7 const user = await users.load(userId) // (2) ~15 ms await. The gap is now8 // wide enough for thousands of tasks.9 if (user.hasUsedPromo(promoId)) return { ok: false, reason: 'already-used' }10 11 // <-- GAP 2: another task may redeem for this same user12 13 promo.remaining -= 1 // (3) shared read-modify-write14 await grants.insert({ userId, promoId }) // (4) shared write, different location15 return { ok: true }16}17 18// Three questions to run on this function, in order:19// Q1 Which lines touch state that appears in the invariant? -> 1, 3, 420// Q2 For each, what could another task do immediately after? -> see below21// Q3 Does any later line assume something an earlier line read? -> yes: line 322// assumes line 1's reading of remaining > 0 is still true.23//24// The invariant mentions grantsIssued and promo.remaining. Line 3 and line 425// update those two facts non-atomically, across an await. There is therefore a26// schedule in which remaining is decremented and the grant never lands, and a27// schedule in which the grant lands twice.The drill: put a switch after every shared access
Here is the procedure, and it is deliberately mechanical so that it works when you are tired and reviewing someone else's diff at 18:00. Write the function's shared accesses as a numbered list — reads and writes of state that appears in the invariant, nothing else. Then, for each position *between* consecutive accesses, insert a hypothetical task switch and ask one question: if a second copy of this function ran to completion right here, would the invariant still hold when we resume?
That question is answerable without cleverness. It has a yes or a no, and the no comes with the failing schedule already written. The schedule below is the answer for gap 1 in redeem, played out with promo.remaining at 1 — the last redemption — and two tasks in flight.
Two refinements make the drill sharper. First, the second copy does not have to be the *same* function; check for any other writer of the same state, because those are the ones nobody thinks of. Second, remember that a switch can also occur *inside* what looks like one access: promo.remaining -= 1 is itself read-modify-write, so position it as two entries in the list, not one. That is Interleavings: The Schedule Is Part of the Program applied recursively.
| # | Checkout A — user 700 | Checkout B — user 701 | State |
|---|---|---|---|
| 1 | read promo.remaining → 1 | · | remaining=1 grants=99 |
| 2 | check 1 > 0 → true | · | remaining=1 grants=99 |
| 3 | await users.load(700) — task suspends | · | remaining=1 grants=99 |
| 4 | · | read promo.remaining → 1 | remaining=1 grants=99 |
| 5 | · | check 1 > 0 → true | remaining=1 grants=99 ✕ Two tasks have now both been told there is one code left. The invariant is doomed here, several milliseconds before any write. |
| 6 | · | await users.load(701); resume; write remaining = 0 | remaining=0 grants=99 |
| 7 | · | await grants.insert(701) | remaining=0 grants=100 |
| 8 | resume; write remaining = 1 - 1 | · | remaining=0 grants=100 |
| 9 | await grants.insert(700) | · | remaining=0 grants=101 ✕ 101 grants issued against a 100-use promo, and remaining is 0 so the counter looks correct. The overspend is invisible in the counter and visible only in the grants table. |
Running the drill on a diff
The output of the drill should be written down, because "I thought about it" is not reviewable. The annotated listing below is what a completed pass looks like: every shared access numbered, every gap examined, every gap given a verdict, and the fix attached to the gap it closes rather than to the function as a whole.
Notice the last line of the analysis. Two of the three gaps in redeem cannot be closed with a process-local lock at all, because the state lives in the database and there is more than one instance of this service running. That is a genuinely common outcome and it is why the drill is valuable: it tells you not just *that* there is a race but *where the arbiter has to live*. See A Mutex on Server A Does Nothing About Server B and The Database Solves Concurrency For Its Data, Not For Your Memory.
redeem(promoId, userId) invariant: grantsIssued + remaining === 100
and remaining >= 0
shared accesses (only state named in the invariant):
S1 read promo.remaining line 2
S2 read promo.remaining line 9 (the RMW's read half)
S3 write promo.remaining line 9 (the RMW's write half)
S4 write grants line 10
gaps, and what a second task can do in each:
S1 -> S2 width: one await (~15 ms)
second task can: pass its own check, decrement, insert a grant
verdict: RACE. Both tasks act on remaining > 0 read before either wrote.
evidence: schedule above. Overspend scales with in-flight requests.
S2 -> S3 width: nanoseconds (no await between them)
second task can: complete its own read-modify-write
verdict: RACE. Classic lost update; see [[interleavings]].
note: narrow enough to pass every test and still fire in production.
S3 -> S4 width: one await (~8 ms)
second task can: nothing that breaks the invariant...
...but a CRASH here does: remaining is decremented, no grant issued.
verdict: NOT a race, but a partial-failure hole. A code is burned
and nobody receives it. Needs the two writes in one transaction.
fixes, attached to the gap each one closes:
S1->S2, S2->S3 move the decision into the store, atomically:
UPDATE promos SET remaining = remaining - 1
WHERE id = ? AND remaining > 0
-- then check affected rows. 0 rows means exhausted.
Closes both gaps; works across service instances.
S3->S4 put the decrement and the grant insert in one transaction.
Closes the partial-failure hole. Does NOT close S1->S2.
NOT a fix a process-local mutex around lines 2-10. It closes the gaps
for one instance and closes nothing for the other five
replicas, and it now holds a lock across two awaits.
See [[lock-scope]], [[local-lock-not-distributed]].Key points
- A race condition is logical correctness that depends on timing. It is a property of the *design*, not of the memory model — see Data Race Is Not Race Condition for the other thing that word gets used for.
- Nearly all of them are check-then-act: a fact is read, a decision is made from it, and the fact is allowed to change before the decision is applied.
- The check produces a fact about the past. Treating it as a fact about the present is the bug, stated in one sentence.
- The method: number the shared accesses named in the invariant, insert a hypothetical switch in every gap, and ask whether a complete second execution there would break the invariant.
- Count read-modify-write as two accesses. Half the races hide inside what looks like a single statement.
- An
awaitin a gap widens it by six orders of magnitude. Look at those gaps first. - The drill also tells you *where* the fix must live — if the state is in a database and the service has replicas, no in-process primitive can close the gap.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • State the invariant, and list only the shared locations it mentions — this bounds the analysis and keeps it finishable.
- • Number every read and every write of those locations in source order, splitting read-modify-write into its two halves.
- • For each gap between consecutive accesses, hypothesise a task switch and run a complete second execution of any writer of that state.
- • Evaluate the invariant on resume. If it can be false, you have the failing schedule and should write it down as evidence.
- • Attach each candidate fix to the specific gap it closes, and check that it closes that gap for every instance of the program, not just this one.
- • A checks remaining=1; A awaits; B checks remaining=1; B decrements and grants; A resumes, decrements to 0 and grants. 101 grants against a 100-use promo, with the counter reading a tidy 0.
- • A checks; A decrements; A grants; B checks remaining=0 and rejects. Correct — and it is the only schedule a single-request test can produce.
- • A decrements to 0; A crashes before inserting the grant. Not a race — a partial failure. The code is consumed and never delivered, which needs a transaction, not a lock.
- • Both tasks pass
user.hasUsedPromobefore either grant lands, so one user receives the same promo twice. A second race in the same function, on different state, found by the same drill. - • With
UPDATE ... WHERE remaining > 0: A's update affects 1 row, B's affects 0. B rejects. No interleaving breaks the invariant, because the check and the act are one statement executed by one arbiter.
- • The drill guarantees you have examined every gap you listed. It does not guarantee you listed every shared access — state reached through a cache, a closure or a library callback is routinely missed.
- • A local mutex guarantees mutual exclusion within one process. It guarantees nothing when a second replica of the service is running, which is the normal deployment.
- • A conditional
UPDATE ... WHEREguarantees atomicity of check-and-act at the row, under the database's isolation level — and guarantees nothing about a second, related row unless both are in one transaction. - • Passing tests guarantee that the schedules the harness produced were correct. Under a scheduler you do not control, that is a statement about a sample, not about the program.
- • The drill itself costs nothing at runtime; the fixes do. Moving the decision into the database converts an in-process race into a row-level lock and therefore into database contention on a single hot row. See The Database Solves Concurrency For Its Data, Not For Your Memory, and the hot-row contention it creates.
- • A promo counter is a single hot row by construction — every checkout touches it — so the last minutes of a popular promo are a contention event as well as a correctness one.
- • The local-mutex non-fix has the worst contention profile of all the options: it serialises the handler across two awaits while still being incorrect across replicas.
- • Overspend — more grants issued than the limit allows, scaling with in-flight concurrency rather than by one.
- • Lost update on the counter, hidden by arithmetic: two tasks write the same value from different stale reads, so the counter looks right.
- • Duplicate grant to one user, from the second check-then-act in the same function.
- • Partial failure between the decrement and the grant — not a race, but found by the same pass and equally invisible.
- • False confidence: a local lock is added, the code review passes, and the bug survives because the service runs six replicas.
- • On every diff that touches shared state — the pass takes two minutes on a small function and is the only review technique that reliably finds this class.
- • During incident analysis, where the drill turns "it must be a race somewhere" into a specific gap with a specific schedule you can show people.
- • Before choosing a primitive: knowing which gaps must close tells you whether you need a lock, a transaction, an atomic, or a redesign.
- • When applied to state that is not actually shared — the analysis is real work and produces nothing.
- • When it turns into exhaustive enumeration for more than two or three actors. Past that, use a race detector or a model checker; hand analysis stops being reliable. See Race Detectors: What They Find, and What They Structurally Cannot.
- • When it produces a defensive lock at every gap. Some gaps are benign — a stale read used only for a metric does not need closing, and closing it costs contention forever.
- • Compare two independently derived counts:
promo.remainingagainstSELECT count(*) FROM grants WHERE promo = ?. Divergence is the direct evidence. - • Alert on the invariant, not the counter. The counter was correct in the failing schedule above; the comparison was not.
- • Reproduce deliberately: fire N concurrent redemptions against a promo with 1 remaining and assert exactly one succeeds. Repeat under an artificial delay inserted at each gap. See Stress Testing: A Test That Passed Once Proves Nothing.
- • Check the affected-row count of every conditional update. Code that issues
UPDATE ... WHERE remaining > 0and ignores the row count has implemented the fix and thrown away its result.
- • The analysis is cheap; the fixes are not. Moving the arbiter into the database couples the handler to transaction semantics and isolation levels the team must now understand.
- • Every gap closed with a lock adds an acquisition order and a hold-time budget to the module's contract.
- • Recording the analysis — which gaps were examined and which were judged benign — is extra documentation that nothing enforces, and its absence is why the same race is re-introduced two quarters later.
- • Make the check and the act one statement executed by an arbiter: a conditional
UPDATE ... WHERE, an atomicputIfAbsent, anINCRwith a bound. No gap means no analysis. See The Database Solves Concurrency For Its Data, Not For Your Memory. - • Make the operation idempotent and let it run twice safely — a unique constraint on
(userId, promoId)turns the duplicate-grant race into a caught constraint violation. See Optimistic Concurrency Control. - • Serialise the decision onto one owner: a single task, partition or worker that owns this promo. Removes the interleaving instead of reasoning about it. See The Actor Model.
- • Accept the race where the invariant is soft. A promo that may overspend by a fraction of a percent under burst may be a business decision, not a bug — but that must be a decision someone made, not an accident.
if (balance >= 100) withdraw(100) — drive it until it overdraws
balance = 100
withdraw(amount): # both tasks run this concurrently
b = read(balance) # 1
if b >= amount: # 2 <- decided on a value that may already be stale
debit(amount) # 3| # | Withdrawal A (100) | Withdrawal B (100) | State |
|---|---|---|---|
| 1 | rA ← read balance | · | balance=100 paidOut=0 |
| 2 | if rA >= 100 | · | balance=100 paidOut=0 |
| 3 | debit 100 | · | balance=0 paidOut=100 |
Two increments, twenty schedules: find the one that loses an update
| # | Task A — counter++ | Task B — counter++ | State |
|---|---|---|---|
| 1 | rA ← counter | · | counter=0 rA=0 rB=0 |
| 2 | rA ← rA + 1 | · | counter=0 rA=1 rB=0 |
| 3 | counter ← rA | · | counter=1 rA=1 rB=0 |
| 4 | · | rB ← counter | counter=1 rA=1 rB=1 |
| 5 | · | rB ← rB + 1 | counter=1 rA=1 rB=2 |
| 6 | · | counter ← rB | counter=2 rA=1 rB=2 |
Race detector lab
| task | access | location | holding |
|---|---|---|---|
| A | read | count | — |
| A | write | count | — |
| B | read | count | — |
| B | write | count | — |
| # | Task A | Task B | State |
|---|---|---|---|
| 1 | r1 = count | · | count=0 done=0 |
| 2 | · | r2 = count | count=0 done=0 |
| 3 | count = r1 + 1 | · | count=1 done=1 |
| 4 | · | count = r2 + 1 | count=1 done=2 ✕ count equals the number of increments that have completed — broken here |
What people believe, and what is true
It is a race condition only if two threads are involved.
Two *tasks* are enough. A single-threaded event loop interleaves at every await, and the resulting race is identical in kind and wider in window.
I checked it right before using it, so it is fine.
"Right before" is the entire problem. The check yields a fact about the past; any gap at all, however short, is a gap.
Adding a mutex fixed it — the test passes now.
It fixed it for one process. If the state lives in a database and the service has replicas, the mutex closes the gap on one replica and leaves it open on the others.
The counter is correct, so we did not overspend.
In the schedule above the counter is exactly 0 and 101 grants were issued. Detecting this class requires comparing two independently derived values.
Go deeper
Overview
A race condition is a bug whose presence depends on timing. Almost all of them are: check something, then act on it, with a gap in between.
Practical
Number the shared accesses, examine each gap, ask whether a full second execution there breaks the invariant. Write the answer down and attach a fix to each gap that fails.
Advanced
Split read-modify-write into two accesses, count writers you did not write, and check whether the fix works for every instance of the program. Judge benign gaps explicitly rather than closing them by reflex — a defensive lock in a benign gap is permanent contention bought for nothing.
Internals
Hand enumeration is exponential in the number of actors and accesses, which is why the tooling exists: dynamic race detectors instrument accesses and track a happens-before relation (Happens-Before: The Edge That Makes a Write Visible), while model checkers explore the schedule space exhaustively for small programs. Neither finds a logical race whose accesses are all properly synchronized — the double-booking in Finding the Critical Section is invisible to both.