Concurrency Cheat Sheet
Need X → use Y, with the reason and the price. Every row carries a cost column on purpose: concurrency is always bought with complexity, and a recommendation that does not say what it costs is not a recommendation.
Choosing a model
| Need | Use | Why | Cost |
|---|---|---|---|
| The work is waiting on network or disk | Async tasks on one thread, with a concurrency limit | Waiting costs a suspended task, not a thread stack — thousands overlap on one core | Every CPU-heavy line in a handler now stalls everyone else on that loop |
| The work is burning CPU | Processes, or threads in a language that runs them in parallel | Only real parallelism reduces wall-clock time for compute; async just interleaves it | Copying data across a process boundary, and a startup cost per worker |
| Mixed: heavy compute inside a request handler | Async at the edge, a bounded worker pool behind it | Keeps the loop free for I/O while compute runs somewhere it cannot block anyone | Two schedulers to reason about, and a queue between them that can fill |
| CPython and the work is compute-heavy | Processes (or a native extension that releases the GIL) | CPython threads do not execute Python bytecode simultaneously; processes do | Serialization on every argument and result, and per-process memory |
| JavaScript and the work is compute-heavy | Worker threads (Node) or web workers (browser) | The main loop is single-threaded; workers are the only way to get another core | Message-passing only — no shared objects unless you reach for SharedArrayBuffer |
| Fewer than a few hundred milliseconds of total work | Do it sequentially | Below the coordination overhead, concurrency is pure loss and pure risk | You will be asked why it is "not parallel"; the answer is a measurement |
| Strong isolation between units of work | Separate processes | A crash, a leak or a runaway allocation stays inside one address space | IPC, slower startup, and no cheap sharing of large data |
Shared state
| Need | Use | Why | Cost |
|---|---|---|---|
| Two tasks touch the same variable | First ask whether they need to at all | The cheapest synchronization is the sharing you removed | Copying or partitioning costs memory, and sometimes a merge step |
| The value never changes after construction | Make it immutable and publish it safely | Readers of an immutable object need no synchronization at all | Every update allocates a new object; churn moves to the allocator and GC |
| Mostly reads, occasional whole-state replacement | Copy-on-write snapshot swapped behind one atomic reference | Readers never block and never see a half-written state | Writes copy the whole structure, and readers may hold a stale snapshot |
| You cannot say what "correct" means here | Stop and write the invariant as a sentence | A primitive chosen without an invariant protects a region, not a property | It is slow, unglamorous thinking, and it usually shrinks the design |
| "Is this a race?" | Enumerate interleavings at the granularity of individual reads and writes | A race is found by exhibiting a schedule, not by staring at the code | Tedious, and the interesting schedules are rarely the first ones you try |
| Unsynchronized access in C++ where one side writes | Fix it — this is a data race, not just a race condition | A data race is undefined behaviour: the compiler may assume it cannot happen | The fix is a lock or an atomic, and both cost something on the read path |
count += 1 from two tasks | An atomic increment, or a lock around the whole update | Read-modify-write is three steps; the increment you lose is one that interleaved | An atomic is one operation only — it does not protect the next line |
Synchronization
| Need | Use | Why | Cost |
|---|---|---|---|
| One invariant spanning several fields | A single mutex covering exactly those fields | Mutual exclusion is what keeps a multi-field update from being observed half-done | Every reader now waits, and every path through the region is serialized |
| A lock is held for a long time | Shrink the region: compute outside, mutate inside | Critical-section length is the multiplier on every waiting thread | More code runs unsynchronized, so the boundary needs more care |
Locks held across an await or an HTTP call | Never do this — release before the I/O, revalidate after | A lock held for a network round trip converts a fast path into a queue | Revalidation after reacquiring is extra code, and it is easy to get wrong |
| Many readers, rare writers, long read sections | A read/write lock | Readers proceed together; only writers exclude | More state, worse worst-case, and writer starvation unless the lock is fair |
| Cap how many things do X at once | A semaphore with N permits | A semaphore counts a resource; a mutex only says "one" | A permit not released on the error path leaks capacity permanently |
| Wait until a condition becomes true | A condition variable, waited on inside a `while` loop | Wakeups are hints, not proofs — the predicate must be rechecked | Signal before wait and the waiter sleeps forever: a lost wakeup |
| "It woke up but the condition was false" | Recheck the predicate and wait again — that is the contract | Spurious wakeups are permitted by the platform, not a bug in your code | The loop is mandatory boilerplate on every wait site |
| Recursive call re-enters a locked region | Restructure so the lock is taken once, at the boundary | A non-reentrant mutex deadlocks against itself; a reentrant one hides the design flaw | Splitting locked and unlocked variants of a function duplicates signatures |
Queues & backpressure
| Need | Use | Why | Cost |
|---|---|---|---|
| Fast producer, slow consumer | A bounded queue between them | The bound is what turns "unlimited memory growth" into "the producer waits" | The producer now blocks, so it needs an answer for what to do while blocked |
| The queue is unbounded "for safety" | Bound it and decide the overflow policy explicitly | Unbounded means the failure mode is an OOM at 3 a.m. instead of a rejection now | Someone must choose block, drop-oldest, drop-newest or reject — and own it |
| Handing work between stages | A channel, sized deliberately | Ownership transfer replaces shared mutable state with a copy and a handoff | Copying cost per item, and a fixed pipeline shape that is harder to change |
| Independent entities with their own state | An actor with a mailbox | One actor processes one message at a time, so its state needs no lock | Everything becomes asynchronous, and a slow actor is now a hidden bottleneck |
| Avoid locks entirely for a shared structure | Give one owner the data and send it messages | No sharing means no interleaving to reason about — the invariant is local | Latency of a round trip, and the owner is a serialization point by design |
| Shut down while work is in flight | Stop intake, drain the queue with a deadline, then cancel the rest | Ordering matters: closing the input first is what makes draining finite | A drain deadline means some work is abandoned; it must be safe to retry |
| Queue depth looks fine but latency is bad | Measure queue *age*, not depth | A steady depth with rising age means arrivals exceed service rate | Age requires an enqueue timestamp on every item |
Async
| Need | Use | Why | Cost |
|---|---|---|---|
| Awaiting independent calls one after another | Start them all, then await the collection | Sequential awaits sum latencies that could have overlapped | Now all of them run at once — against a dependency that may not want that |
Promise.all over 10,000 items | A bounded map with a concurrency limit | Promise.all starts everything immediately; the array is a schedule, not a limit | A limit means the total takes longer — deliberately |
One rejected promise in an all | Decide between all-or-nothing and per-item results | Promise.all rejects on the first failure while the others keep running unwatched | Collecting settled results means handling a mixed success/failure array |
| A handler does 200ms of JSON work | Move it to a worker, or chunk it and yield | On a single event loop, that 200ms is added to everyone else's latency | Workers add serialization; chunking adds interleaving you must make safe |
| "Make it async so it runs in parallel" | Say which core the second thing would run on | Async gives overlap of waiting, not simultaneous execution of computation | None — but the conversation is where the misunderstanding usually surfaces |
| Async code has to hold a lock | Use an async-aware lock, never a blocking one | A blocking mutex on an event loop parks the thread that would run the holder | Async locks are slower and do not protect against blocking code elsewhere |
| A task nobody awaits | Give it an owner with a scope, or do not start it | An unawaited task swallows its exception and outlives the request that made it | Scoping means the parent waits, which sometimes is exactly what you were avoiding |
Pools & limits
| Need | Use | Why | Cost |
|---|---|---|---|
| Spawning a thread per unit of work | A pool with a fixed size and a queue | Threads are a bounded resource; a pool makes the bound explicit and visible | Work now queues, and the queue needs a bound and a policy of its own |
| "How many threads should the pool have?" | Derive from what each task waits on, then measure | There is no universal formula — it depends on wait fraction, memory and downstream limits | Measuring means load-testing with a realistic mix, which takes real time |
| All workers busy, work still arriving | Decide: queue, shed, or scale — before it happens | Saturation is not a bug; the absence of a policy for it is | Shedding returns errors to users; queueing raises latency for everyone |
| Uneven task durations across workers | A work-stealing scheduler | Idle workers take from busy queues instead of waiting for a rebalance | Steal attempts contend, and locality suffers when a task moves cores |
| Client concurrency exceeds server capacity | Bound it on the client side too | Sending 500 concurrent requests to a service sized for 50 makes both sides slower | You give up peak throughput you could not actually sustain anyway |
| A pool per subsystem, all sharing a database | Add up the limits before trusting any of them | Four pools of 25 is 100 connections against a database configured for 60 | Central accounting of limits is organizational work, not just code |
| A cache entry expires and 1,000 callers miss | Coalesce them into one in-flight computation | Single-flight turns a stampede into one call plus 999 waiters | The waiters share a fate: one slow computation delays all of them |
Parallel decomposition
| Need | Use | Why | Cost |
|---|---|---|---|
| A big independent computation over a collection | Split, run, combine — fork/join | Independence is the whole precondition; if it holds, the decomposition is mechanical | Split and merge are real work, and they are on the critical path |
| Combining results from parallel chunks | A reduction with an associative operator | Associativity is what lets the tree be reordered without changing the answer | Floating-point addition is not associative — the result changes with the shape |
| "How much faster can this get?" | Measure the serial fraction first | Amdahl: 5% serial caps you at 20x no matter how many cores you buy | The measurement itself takes effort, and the answer is often disappointing |
| Tasks depend on each other | Draw the dependency graph and find the span | The longest chain is the floor on runtime regardless of core count | Building the graph forces you to make implicit ordering explicit |
| The same arithmetic on many numbers | SIMD, or a library that already uses it | One instruction over a vector is parallelism inside a single core | Branchy or irregular data defeats it; alignment and layout become your problem |
| Stages that must run in order, on a stream | A pipeline, one worker per stage | Different items occupy different stages at the same time | Throughput is set by the slowest stage, and latency per item goes up |
| Parallelising it made it slower | Compare the chunk size against the coordination cost | Below a threshold, split, schedule and merge cost more than the work | Finding the threshold is a measurement per machine, not a constant |
Failure
| Need | Use | Why | Cost |
|---|---|---|---|
| Two locks taken in two orders | Impose a global lock order and document it | Removing circular wait removes deadlock structurally, not probabilistically | The order becomes a codebase-wide invariant that every new lock must respect |
| Cannot impose an order | Use try-lock with a timeout and a full backoff-and-retry | Breaking hold-and-wait is the other structural fix available to you | You must be able to release everything and redo the work — some code cannot |
| Everything is running, nothing progresses | Look for livelock: retries that keep colliding | Threads that all back off identically re-collide forever — that is not a deadlock | The fix is randomized backoff, which makes timing non-reproducible |
| One thread never gets the lock | Use a fair queueing lock — and accept it is slower | Barging locks favour whoever is cache-hot, which starves the unlucky | Fairness costs throughput; the handoff prevents the fast path |
| A low-priority holder blocks a high-priority waiter | Priority inheritance, if the platform offers it | The holder must be able to finish, so it needs the waiter's priority | Platform-specific and hard to observe; not available in most runtimes |
| Threads pile up behind one lock | Split the lock, shard the data, or remove the sharing | A convoy forms when service time under the lock exceeds the arrival gap | Sharding multiplies locks and reintroduces multi-lock ordering problems |
| Adding threads made throughput drop | Count runnable threads against cores | Past saturation, extra threads add context switches and cache pollution, not work | Reducing thread count feels like giving up capacity; the graph says otherwise |
| A tight loop polls a flag | Block on a condition variable instead | Busy-waiting burns a core to save a few microseconds of wakeup latency | Blocking costs a syscall and a wakeup on every handoff |
Debugging
| Need | Use | Why | Cost |
|---|---|---|---|
| The bug vanishes when you add logging | Treat it as a heisenbug: change the schedule deliberately | Logging changes timing, so the disappearance is evidence, not a cure | Reproducing needs stress and injected delays, which take time to build |
| Suspected data race in C++ or Go | Run it under a race detector in CI | Detectors find unsynchronized conflicting accesses on paths that did execute | Large slowdown and memory overhead, and no proof about paths not taken |
| Everything is slow, CPU is low | Look at lock wait time, not utilization | Waiting does not show up as CPU — the queue is invisible in a CPU graph | You must instrument acquisition sites to get the number at all |
| The process is wedged | Take a thread dump and build the wait-for graph | A cycle in the graph is a deadlock; you can read it off the stacks | Dumps are a point-in-time sample and can be huge and hard to read |
| An async runtime is wedged | Dump pending tasks and their await points | Async stalls have no thread stacks to read — the task inventory is the evidence | Requires runtime support and instrumentation you may have to add first |
| A rare interleaving must be reproduced | Stress test with injected delays and randomized scheduling | Normal runs explore a narrow slice of the schedule space | A green stress run proves nothing about the schedules it did not try |
| Need the exact failing run back | Record and replay it deterministically | Deterministic replay makes a timing bug into an ordinary debugging session | Recording overhead, and tooling that only exists on some platforms |
| Nothing tells you concurrency is degrading | Emit in-flight count, queue depth, queue age and lock wait as first-class metrics | These four move before the outage does; latency moves after | Instrumentation cost, and four more series to store and alert on |