Race Conditions
counter++ is three instructions — load, add, store — and if two threads interleave between them one increment is lost; races are timing-dependent by nature, and the same read-then-act shape on the file system (TOCTOU) is a security bug rather than a counting bug.
The problem
counter++ once, starting from 0. The answer is sometimes 2 and sometimes 1. The source code is one line; the CPU sees three; the scheduler can switch between any of them.counter++ is not one thing
The line counter++ compiles to a read-modify-write: load the value from memory into a register, add one, store the register back. On x86-64 without optimisation that is literally mov eax, [counter] / add eax, 1 / mov [counter], eax. Each thread has its own registers (Context Switching saves and restores them), so two threads executing the sequence have two private copies of the intermediate value. Memory is updated only at the store.
Interleave them. A loads 0. B loads 0. A adds, stores 1. B adds — its register still holds 0 — stores 1. Two increments, final value 1. The scheduler did not have to switch in the middle; on two cores the threads run at the same time and the loads simply happened before either store. The window is a few nanoseconds wide, so at low contention it is hit rarely — one in a million increments — which is exactly what makes the bug survive review and testing and show up as a counter that is 0.0001% low.
Even a single instruction is not safe by itself. inc dword ptr [counter] is one instruction, but on a multi-core machine it is still a load and a store to the memory hierarchy, and another core can slip between them; x86 needs the lock prefix to make it atomic (Atomic Operations). The general rule: any *read then write* of shared state, at any granularity, is a race unless something makes it atomic.
1// counter++ becomes:2// mov eax, [counter] ; load3// add eax, 1 ; modify (in a register private to this thread)4// mov [counter], eax ; store5 6// Thread A Thread B counter7// load -> eax=0 08// load -> eax=0 09// add -> eax=1 010// store <- 1 111// add -> eax=1 112// store <- 1 1 <- one increment lost13 14std::atomic<int> counter{0};15counter.fetch_add(1); // lock xadd: the fix, one indivisible read-modify-writeWhy it is timing-dependent, and why that is the worst property
A race is not wrong; it is *sometimes* wrong. Whether the bad interleaving occurs depends on the scheduler’s decisions, the number of cores, cache state, what else is running, and whether a page fault or interrupt happened to land inside the three instructions. Change any of them — add a log line, run under a debugger, move to a laptop — and the window shifts. Heisenbugs are races almost by definition.
That means the absence of failures is not evidence of correctness. The right tools reason about interleavings rather than sampling them: ThreadSanitizer instruments every memory access and reports two unsynchronised accesses where at least one is a write, with both stack traces, on the first run that exhibits the *access pattern* even if the *bad timing* never occurred; Go’s -race is the same technology; Java has JCStress; Rust rejects the pattern at compile time for plain data. Stress tests with many threads on many cores raise the hit rate but still only sample.
Code review catches races by shape, not by timing: look for shared state, then for every read of it that is followed by a write that depends on the read, and ask what makes that pair indivisible. If the answer is "nothing", it is a race regardless of whether it has ever failed.
- The window is nanoseconds; the failure rate is load-dependent; the symptom is drift, not a crash.
- Sanitizers detect the *pattern* (unsynchronised conflicting accesses), not the *timing*.
- Rate of failure ∝ contention: a race that never fires in dev fires constantly at peak.
Python, JavaScript and the GIL
CPython’s global interpreter lock serialises bytecode execution, so only one thread runs Python at a time — and people conclude counter += 1 is therefore safe. It is not. The statement compiles to several bytecodes (LOAD_GLOBAL, LOAD_CONST, BINARY_OP, STORE_GLOBAL), and CPython may switch threads between any two of them (every 5 ms by default, or at any blocking call). Two threads incrementing a global a million times each routinely end below two million. The GIL protects the interpreter’s own data structures; it does not make your compound operations atomic. Free-threaded builds (3.13+, --disable-gil) remove the GIL and make this more visible, not less. Use threading.Lock, or a queue.Queue, or itertools.count() which increments in C under one bytecode.
JavaScript runs one thread per realm, and code between two awaits cannot be interleaved with other JavaScript in that realm — so counter++ on a plain number is safe from *thread* races. Two things reintroduce them. SharedArrayBuffer shared with Web Workers or worker_threads is real shared memory, and i32[0]++ on it is the same three-instruction race; Atomics.add is the fix. And logical races across await points: if (!cache.has(k)) { const v = await fetch(k); cache.set(k, v) } runs the check, yields, and by the time it resumes another call has done the same — a duplicate fetch, or a lost update on whatever the two callers write next. Single-threaded does not mean race-free; it means the interleaving points are the awaits.
1import threading2n = 03def work():4 global n5 for _ in range(1_000_000):6 n += 1 # LOAD_GLOBAL, LOAD_CONST, BINARY_OP, STORE_GLOBAL — switchable between any two7 8ts = [threading.Thread(target=work) for _ in range(4)]9[t.start() for t in ts]; [t.join() for t in ts]10print(n) # typically far below 4_000_000 on CPython 3.9–3.1211 12# fix: a lock around the read-modify-write13lock = threading.Lock()14# with lock: n += 1TOCTOU: the race on the file system
The second family of races has one thread and one adversary. Time-of-check to time-of-use: a program checks something about a path — access("/tmp/report", W_OK) says the user may write it, stat says it is a regular file, exists says it is absent — and then acts on the path. Between the check and the act, another process replaces what the path points to: swaps in a symlink to /etc/passwd, creates the file first, moves a directory. The check was true; the action is now against a different object. Because the name is only a pointer to an object (Files, Paths and Names), any decision made about a *name* can be invalidated before the *object* is opened.
This is a security bug more often than a correctness bug: the classic privilege escalation is a setuid program that checks a user-controlled path and then opens it as root. The fix is never "check faster". It is to make the check and the use one operation on the *object*: open first, then fstat the descriptor you hold; create with O_CREAT | O_EXCL so the create fails if the file already exists instead of testing exists first; use O_NOFOLLOW to refuse symlinks; use openat with a directory descriptor so the directory cannot be swapped under you; use mkstemp instead of guessing a temporary name. Every one of these replaces a check-on-name with an atomic operation-on-object.
The pattern generalises to any shared resource with a name: "if the row does not exist, insert it" is a TOCTOU against a database unless the insert is ON CONFLICT-safe or the read is locked (Isolation Levels); "if the key is absent, set it" in Redis is the reason SETNX exists; "if the port is free, bind it" is why you bind and handle EADDRINUSE rather than probing.
1// racy: someone can create or symlink /tmp/out between the two calls2if (access("/tmp/out", F_OK) != 0)3 fd = open("/tmp/out", O_WRONLY | O_CREAT, 0600);4 5// atomic: the kernel checks existence and creates in one operation on the directory6fd = open("/tmp/out", O_WRONLY | O_CREAT | O_EXCL | O_NOFOLLOW | O_CLOEXEC, 0600);7if (fd < 0 && errno == EEXIST) { /* someone was first; decide, do not retry blindly */ }The same race in the database
Replace "thread" with "transaction" and "counter" with "row" and nothing changes. Two transactions SELECT balance (100), each compute a new value in the application, each UPDATE balance = new_value; the second write overwrites the first. That is the lost update anomaly, and READ COMMITTED — the default in PostgreSQL — permits it, because the read and the write are two statements with a gap between them, exactly like the load and the store. UPDATE accounts SET balance = balance - 10 is the database’s fetch_add: one statement, read-modify-write inside the engine under a row lock. SELECT … FOR UPDATE is the database’s mutex. Serialisable isolation is the database detecting the interleaving and aborting one side, which is what a CAS loop does in memory.
Understanding the OS-level race is what makes the database-level remedies obvious rather than folklore: every isolation level and every locking clause is a statement about which interleavings the engine will prevent for you, and the ones it will not prevent are yours to prevent with the same tools.
Key points
counter++is load, add, store; two threads interleaving between load and store lose an increment. Any read-then-write of shared state is a race unless something makes it atomic.- Races are timing-dependent: rare at low load, frequent at peak, and shifted by any change to the environment. Absence of failures proves nothing.
- ThreadSanitizer and Go’s race detector find the access pattern without needing the bad timing; use them in CI.
- The CPython GIL does not make
+=atomic; JavaScript is race-free only betweenawaits and only withoutSharedArrayBuffer. - TOCTOU is a race between a check on a name and an action on the object; fix it by operating on descriptors and using atomic flags (
O_EXCL,O_NOFOLLOW,openat,mkstemp), never by checking faster. - The database lost update is the same race;
SET n = n + 1,FOR UPDATEand serialisable isolation are the database’s atomic RMW, mutex and CAS.
Why does this exist?
Mechanisms are answers to constraints. Open each question before reading the answer.
▸Why can’t the compiler just make `++` atomic?
It could — that is std::atomic and the lock prefix — but it costs a cache-line lock and a fence on every increment, roughly 5–20× a plain add and far worse under contention. The language makes you ask for it so that the common single-threaded case stays cheap.
▸Why are races so hard to reproduce?
The failing window is a few instructions wide and whether two threads land in it together depends on the scheduler, core count, caches and interrupts. Anything you do to observe it — a breakpoint, a print — changes the timing that produced it.
▸Why does a TOCTOU bug need an attacker to matter?
Because the window between check and use is usually tiny and the program itself will not swap the file. But a hostile local process can hammer the swap in a loop until it wins once, and once is enough when the program runs with more privilege than the attacker.
Race condition: counter++
How it fails
What the failure looks like from inside real software.
- Analytics counters drift a fraction of a percent below the true value and nobody can reproduce it locally.
- A Python service that increments a shared dict from a thread pool reports impossible totals; the author assumed the GIL protected it.
- An async JavaScript cache fetches the same key twice under load: two callers passed the
hascheck before eitherset. - A setuid helper checks
access()on a user-supplied path and then opens it; a symlink swap lets the user read root-owned files. - A temp file created with a predictable name after an
existscheck is pre-created by another user; the program writes into their file. - Two concurrent HTTP requests both read a row, both compute, both write; one customer’s order quantity is silently overwritten (a database lost update at READ COMMITTED).