Concurrency Patterns
The shapes that already have known answers — and the failure modes that come with each one. A pattern is not a recommendation: it is a problem, a structure, the situations where it genuinely helps, and the specific ways it goes wrong. All four are shown together, because a pattern learned without its failure modes is a pattern you will apply where it does not belong.
Module
Producer / Consumer
PipelinesThe problem · Something generates work faster than something else can process it — a crawler filling a parse queue, a socket reader feeding a decoder, a webhook endpoint feeding a job runner. Right now they are the same function, so the fast half runs at the speed of the slow half.
Structure
- A bounded queue sits between the two sides. Producers enqueue; consumers dequeue. Neither holds a reference to the other.
- The queue is the only shared mutable state, and it is the only thing that needs synchronization — the items themselves are handed over, not shared.
- Both sides block on the queue: producers when it is full, consumers when it is empty. That blocking is the pattern, not an inconvenience.
- Shutdown is explicit: close the input, let consumers drain until empty-and-closed, then stop.
When it is genuinely useful
- The two stages have genuinely different rates, and you want the fast one to keep working during the slow one's bursts.
- You want to scale consumers independently — three parsers for one reader.
- The handoff is a natural place to add a metric: depth, age, enqueue rate, service rate.
- You want an explicit place where overload becomes visible instead of a place where memory quietly grows.
Trade-offs
- The queue adds latency per item even when nothing is contended, and it adds a copy or an ownership transfer.
- The bound must be chosen. Too small and a normal burst blocks the producer; too large and stale work is served long after anybody wanted it.
- Two components now have a shared lifecycle, and getting shutdown ordering wrong hangs the process.
- Errors on the consumer side are no longer on the producer's stack, so error propagation has to be designed rather than inherited.
How it fails
- Unbounded queue plus a persistently slow consumer: memory grows until the process is killed, and the last thing in the log is unrelated.
- Consumer dies silently; the queue fills, producers block forever, and the system looks idle rather than broken.
- Shutdown deadlock: the producer waits for space in a full queue while consumers have already been told to stop.
- Poison item — one message that always throws — is retried forever at the head of the queue and starves everything behind it.
- Queue depth looks healthy while item *age* climbs, because arrival rate has quietly exceeded service rate.
Thread Pool
PoolsThe problem · Every request currently starts a thread. It works at 50 requests per second and falls over at 500 — not because the machine is out of CPU, but because it is out of stacks and the scheduler is spending its time switching between threads that are all waiting.
Structure
- A fixed number of worker threads are created once and reused. Each loops: take a task, run it, take the next.
- A bounded queue holds submitted tasks. The pool size caps concurrency; the queue bound caps the backlog.
- Submission returns a future or a handle so the caller can await the result, or decide not to.
- A rejection policy is chosen up front for the case where the queue is full: block the caller, run it on the caller's thread, or reject.
When it is genuinely useful
- Tasks are short relative to thread creation cost, and there are many of them.
- You want a hard, visible ceiling on how much work is in flight.
- The pool boundary is a natural place to attach a metric, a timeout and a name for the workload.
- Different workloads should not compete: separate pools give separate ceilings and separate failures.
Trade-offs
- Pool size is a decision with no universal formula — it depends on how much of each task is waiting, on memory per task, and on downstream limits.
- Work now queues, so latency has two components: waiting for a worker and running.
- Thread-local state persists between unrelated tasks, which is either a useful optimization or a subtle leak.
- A pool sized for one workload becomes the wrong size the moment a second workload is added to it.
How it fails
- Pool exhaustion: every worker is blocked on something slow, and unrelated fast tasks queue behind them.
- Pool deadlock: a task submits a subtask to the same pool and waits for it, while all workers are doing exactly that.
- Unbounded queue in front of a bounded pool — the ceiling on threads is real, the ceiling on memory is not.
- Silent task failure: an exception in a worker is swallowed by a future nobody inspects, and the work simply never happened.
- Blocking calls inside a pool sized for CPU work turn N cores into N waiters.
Lessons behind this pattern
Worker Pool over a Queue
PoolsThe problem · A backlog of ten thousand jobs has to be processed by a fleet that can restart, scale and lose members. Unlike an in-process thread pool, the work has to survive the worker.
Structure
- Jobs live in a durable queue outside the workers. Workers are interchangeable and stateless.
- Each worker claims a job with a visibility timeout or a lease, processes it, then acknowledges it.
- An unacknowledged job returns to the queue after the lease expires, so a crashed worker loses nothing but time.
- Concurrency is worker count times per-worker parallelism — and it is that product, not either factor, that the database sees.
When it is genuinely useful
- Jobs are independent and each one takes long enough that a lease and an acknowledgement are cheap by comparison.
- Throughput must scale by adding processes or machines rather than threads.
- Losing a worker mid-job must not lose the job.
- You need per-job retry and dead-lettering rather than an in-process exception.
Trade-offs
- At-least-once delivery is the norm, so every job must be idempotent — that requirement propagates into your data model.
- The queue is now a component to operate, monitor and pay for.
- Ordering is lost unless you add partitioning, and partitioning reintroduces head-of-line blocking.
- Debugging spans two processes and a broker instead of one stack trace.
How it fails
- Duplicate execution when a lease expires while the job is still running — the classic double-charge.
- Poison job retried forever, consuming the whole fleet's capacity until someone adds a retry limit.
- Aggregate connection exhaustion: 40 workers × 10 connections each against a database configured for 100.
- Scaling workers up simply moves the bottleneck downstream and makes the shared dependency the new failure.
- Silent backlog growth over a weekend because depth was alerted on and age was not.
Fork / Join
ParallelThe problem · One computation over a large input takes 40 seconds on one core while seven cores sit idle. The input can be split, the parts do not talk to each other, and the results can be combined.
Structure
- Split the input into chunks large enough to pay for their own scheduling.
- Fork: hand each chunk to a worker, usually a work-stealing pool.
- Recurse: a chunk still too large splits again, down to a sequential cutoff.
- Join: wait for all children, then combine — and the combine step is itself part of the span.
When it is genuinely useful
- The problem is genuinely divide-and-conquer: merge sort, tree traversal, a reduction over an array.
- Chunks are independent, so no synchronization is needed inside the parallel phase at all.
- The combine operation is associative, which is what lets the tree be reshaped freely.
- Work per chunk is uneven and a stealing scheduler can rebalance without you writing the logic.
Trade-offs
- Split and combine are pure overhead — they exist only because you parallelised.
- The sequential cutoff is a per-machine constant found by measurement, not a number you can reason to.
- Speedup is capped by the serial fraction (Amdahl) and by the span of the dependency tree, whichever bites first.
- Memory traffic goes up: several chunks are hot at once and may not fit in cache together.
How it fails
- Chunks too small: coordination dominates and the parallel version loses to the sequential one, often by a lot.
- Fork/join inside a pool while holding one of its threads — the join waits for a task that can never be scheduled.
- A non-associative combine (floating-point addition) makes results differ run to run, which then gets reported as a data bug.
- One chunk far larger than the others: the join waits for the straggler and seven workers idle.
- Hidden sharing — a memoization cache, a counter, a logger — turns "independent" chunks into a contention point.
Fan-out / Fan-in
Data ParallelThe problem · A page needs data from five services. Called one after another it takes 1.2 seconds; the five calls do not depend on each other, so four of those requests are pure waiting.
Structure
- Issue all independent calls without awaiting each one in turn.
- Await the collection, with a per-call timeout and an overall deadline that is not simply their sum.
- Combine results, deciding in advance which of them are required and which are optional.
- On failure, decide between all-or-nothing and partial results — and if it is all-or-nothing, cancel the ones still running.
When it is genuinely useful
- The calls are genuinely independent and the total is dominated by the slowest, not by the sum.
- Some branches are optional, so a degraded page beats a failed one.
- The downstream services have the capacity to absorb the multiplication in load.
Trade-offs
- Latency becomes the maximum of the branches, which makes tail latency the thing that matters instead of the mean.
- One request in becomes five requests out — a 5x load multiplier on every dependency, at every level of the fan.
- Partial-failure handling is real product design, not error handling: what does the page show?
- Debugging needs a trace; five concurrent spans do not read usefully in a log.
How it fails
- Unbounded fan-out over a list: 500 items becomes 500 simultaneous calls and a self-inflicted outage downstream.
- One slow branch with no timeout holds the whole response, and the client's timeout fires first.
- First rejection abandons the others while they keep running, holding connections nobody will read.
- A retry inside each branch multiplies the fan-out again, turning a blip into a retry storm.
- Nested fan-out: five branches that each fan out five ways is 25 calls from one request, and nobody drew that diagram.
Pipeline
Data ParallelThe problem · Every item must go through decode, transform and write, in that order. The stages cannot be reordered, so it looks sequential — but item 3 can be decoding while item 1 is being written.
Structure
- One stage per transformation, each with its own worker or workers.
- Bounded queues between stages carry items forward and apply backpressure backwards.
- Ownership moves with the item, so no stage shares mutable state with another.
- Throughput is set by the slowest stage; that stage is the only one worth optimizing or widening.
When it is genuinely useful
- A continuous stream of items rather than one batch — throughput matters more than the latency of any single item.
- Stages have different resource profiles: one CPU-heavy, one I/O-heavy, one memory-heavy.
- You want to widen exactly one stage rather than duplicating the whole pipeline.
- Ordering within the stream needs to be preserved and a single-worker stage can enforce it.
Trade-offs
- Per-item latency increases: an item now sits in queues it did not previously exist in.
- Total buffered items — the sum of every queue bound — is memory you must account for.
- The structure is rigid; inserting a stage means rewiring queues and re-tuning bounds.
- A stage that must see the whole stream (a sort, a global aggregate) breaks the pipeline shape entirely.
How it fails
- One slow stage makes every upstream queue full and every downstream stage idle — visible immediately if you graph per-stage depth, invisible if you do not.
- Unbounded inter-stage queues hide the imbalance until memory runs out.
- Widening a stage to more than one worker silently loses ordering that a downstream consumer depended on.
- Shutdown propagated in the wrong direction drops in-flight items or hangs the drain.
- Errors in the middle of a pipeline have nowhere to go unless an error path was designed alongside the data path.
Actor
PipelinesThe problem · A game session, a chat room, a device connection — an entity with its own state that many callers touch concurrently. Locking each field works until you need two of them to change together, and then the lock ordering becomes a project.
Structure
- Each actor owns its state exclusively. No other code holds a reference to it.
- Callers send messages to a mailbox instead of calling methods.
- The actor processes one message at a time to completion, so its state needs no lock at all.
- Replies are messages too — the caller either awaits a future or receives an asynchronous response.
When it is genuinely useful
- State partitions naturally into independent entities that rarely need to change together.
- Per-entity invariants are complex enough that a lock discipline would be error-prone.
- You want the option of moving an actor to another process or machine later without changing its logic.
- Serial processing per entity is acceptable, and often desirable, because it makes behaviour deterministic per actor.
Trade-offs
- Everything becomes asynchronous, including calls that were previously trivially synchronous.
- One actor is a serialization point by construction — a hot actor is a bottleneck with no lock to profile.
- Message passing costs a copy or an ownership transfer per call.
- Operations spanning two actors need a protocol, and that protocol is a distributed-transaction problem in miniature.
How it fails
- Mailbox growth: a slow actor accumulates messages until memory runs out, unless the mailbox is bounded.
- Request/response deadlock: A awaits a reply from B while B awaits a reply from A, and neither is processing.
- A blocking call inside a message handler stalls every message behind it in the mailbox.
- Actors that need to coordinate re-create every problem the pattern was chosen to avoid, now spread across mailboxes.
- Lost messages on restart if the mailbox is in memory and the design assumed it was not.
Lessons behind this pattern
Message Passing
PipelinesThe problem · Two parts of the system need the same data. Sharing it means a lock on every access, and the lock discipline keeps leaking into new code written by people who did not read the comment.
Structure
- Data is transferred rather than shared: the sender gives up access when it sends.
- Communication happens over a channel, a queue or an IPC mechanism with defined delivery semantics.
- Each piece of state has exactly one owner at any moment, so there is no interleaving to reason about.
- Coordination becomes protocol design — which messages, in what order, with what replies.
When it is genuinely useful
- The data is small enough to copy, or the language can transfer ownership without copying.
- You want to eliminate a class of bugs rather than manage it — no sharing means no data races.
- The components may later be separated into processes or machines.
- The team is large enough that "everyone must remember to take the lock" is not a plan.
Trade-offs
- Copying costs CPU and memory; for a large structure touched frequently it can dominate.
- Latency of a round trip replaces a function call.
- Deadlocks do not disappear — they move from lock cycles to message cycles, which are harder to see.
- Debugging follows messages across components rather than frames in one stack.
How it fails
- Cyclic waits between components: A waits for B's reply, B waits for C's, C waits for A's.
- Unbounded channels turning a rate mismatch into unbounded memory growth.
- A "message" that is actually a reference to shared mutable data — the pattern in name only, with all the bugs intact.
- Lost or reordered messages where the delivery guarantee was assumed rather than checked.
- Copy cost discovered late, when the payload grows from kilobytes to megabytes in production.
Lessons behind this pattern
Read/Write Lock
SyncThe problem · A routing table, a feature-flag map, a rate-limit configuration: read on every request, written twice an hour. A plain mutex serializes thousands of readers to protect against a writer who is almost never there.
Structure
- Many readers may hold the lock simultaneously; a writer excludes everyone.
- A writer waits for current readers to finish, and — in a fair implementation — blocks new readers from entering ahead of it.
- Read sections must be long enough that the extra bookkeeping is worth it, and must not upgrade to a write.
When it is genuinely useful
- Reads genuinely dominate — a ratio in the hundreds, not two to one.
- Read sections are long enough that shared access is worth the extra state.
- A single atomically-swapped immutable snapshot is not workable because the structure is large or partially updated.
Trade-offs
- More expensive than a mutex when uncontended, because it maintains a reader count as well as a held flag.
- The reader count is itself a contended cache line — many short reads can perform worse than a plain mutex.
- Fairness policy becomes a decision: reader-preferring starves writers, writer-preferring stalls readers.
- Upgrade from read to write is either unsupported or a deadlock waiting to happen.
How it fails
- Writer starvation under continuous read traffic — the update never lands and nobody notices until the config is hours stale.
- Reader starvation under a writer-preferring policy during a burst of updates.
- Self-deadlock when a read section calls code that tries to take the write lock.
- Chosen for a read/write ratio that was assumed rather than measured, delivering worse performance than the mutex it replaced.
- A reader mutating something reachable through the "read-only" view, which the lock cannot detect.
Semaphore Permit
SyncThe problem · A batch job calls an external API for every row. With ten thousand rows and no limit, the API returns 429s, the connection pool empties, and the retries make both worse. The limit has to exist somewhere; the only question is whether you chose it.
Structure
- A semaphore is created with N permits, where N is the concurrency you have decided to allow.
- Each unit of work acquires a permit before starting and releases it in a
finally— always in afinally. - When no permit is available the task waits, which is the mechanism by which the limit is enforced.
- N is derived from something real: the downstream limit, the memory per in-flight item, or a measured knee in the throughput curve.
When it is genuinely useful
- Bounding in-flight calls to a service with a known capacity or quota.
- Bounding memory by bounding how many large buffers can exist at once.
- Gating a fixed physical resource: file handles, GPU slots, licence seats.
- Adding a limit to existing code without restructuring it into a pool.
Trade-offs
- Peak throughput drops to the throughput you can actually sustain — which is the point, and still needs explaining.
- N is a magic number that will be wrong after the next capacity change unless it is derived and documented.
- A queue of waiters forms in front of the semaphore, and it needs a bound and a timeout of its own.
- Several independently-tuned semaphores against one shared dependency do not compose: their sum is the real limit.
How it fails
- Permit leak on an exception path — capacity ratchets down over days until the system stops entirely.
- Deadlock when a task needs two permits and acquires them in an inconsistent order.
- Waiting without a timeout, so overload turns into an unbounded queue of blocked tasks.
- The limit is per process, so ten replicas apply ten times the limit to a downstream service that saw one number in a design doc.
- A permit held across a retry with backoff, so a slow path occupies capacity while doing nothing.
Barrier
CoordinationThe problem · A simulation runs in rounds. Every worker updates its region, then every worker reads its neighbours' regions. If one worker starts round N+1 while another is still in round N, it reads data from two different rounds and the result is quietly wrong.
Structure
- All participants call
awaiton the barrier at the end of a phase. - Nobody proceeds until the last one arrives; then all are released together.
- A cyclic barrier resets for the next round; a latch is one-shot and never resets.
- The barrier defines a happens-before edge: everything written before it is visible to everyone after it.
When it is genuinely useful
- Phase-structured parallel computation: simulations, iterative solvers, generational algorithms.
- A setup phase must complete on all workers before any of them starts the real work.
- You need a clean point at which to collect metrics or a consistent snapshot.
Trade-offs
- Every phase costs the slowest participant — the barrier converts uneven work into everybody's idle time.
- Participant count is usually fixed, so dynamic scaling needs a different construct.
- Barriers add a synchronization point that memory traffic must cross, so caches cool between phases.
- The structure forces phases even where finer-grained dependencies would allow more overlap.
How it fails
- A participant that fails or returns early leaves everyone else waiting forever — barriers need a failure policy.
- Miscounting participants: one too many and it never trips, one too few and it trips early with work outstanding.
- Load imbalance turning a barrier into a 90%-idle system that still shows 100% CPU on one core.
- Reusing a one-shot latch as if it were cyclic, so the second round passes through instantly.
- Nested barriers with different participant sets deadlocking against each other.
Single-Flight
CoordinationThe problem · A hot cache key expires. In the next 50 milliseconds, 800 requests miss, and all 800 run the same expensive query. The database, which was comfortable a moment ago, is now the outage.
Structure
- An in-flight map keyed by the work's identity records computations currently running.
- The first caller inserts a placeholder and starts the work; later callers find the placeholder and await it.
- When the work completes, everyone waiting receives the same result — or the same error.
- The entry is removed on completion, so the next miss starts a genuinely new computation.
When it is genuinely useful
- Cache stampedes on a small number of hot keys.
- Expensive idempotent computations that many callers request simultaneously.
- Expensive lazy initialization where building it twice is wasteful or unsafe.
- Deduplicating identical outbound requests inside one process.
Trade-offs
- All waiters share one fate: if the single computation is slow, everybody is slow; if it fails, everybody fails.
- The in-flight map is itself shared mutable state and needs its own synchronization — usually a short-lived lock.
- It only coalesces within one process, so ten replicas still produce ten computations for one key.
- A waiter's timeout is now independent of the leader's, which needs care so a timed-out waiter does not orphan the work.
How it fails
- The entry is not removed on the error path, so a failure is cached as an in-flight computation forever.
- Failure amplification: one transient error is returned to 800 callers at once instead of one.
- A key that never quite goes idle keeps one computation permanently in flight and stale.
- Coalescing non-idempotent operations, so a shared result is returned for requests that were not actually identical.
- The map key is not specific enough, and two different requests get each other's answers.
Immutable Snapshot
StateThe problem · Configuration is read on every request by every thread and replaced once a minute by a background refresher. Locking a hot read path to protect against a once-a-minute write is paying continuously for something that almost never happens.
Structure
- The shared state is an immutable object. Nobody ever mutates it in place.
- Readers take the current reference once and use that object for the whole operation — a consistent snapshot by construction.
- A writer builds a completely new object and publishes it with one atomic reference store.
- The store must be a *safe publication*: an atomic or volatile write, so readers cannot observe a partially-constructed object.
When it is genuinely useful
- Read-mostly state where writes are whole-state replacements: config, routing tables, feature flags, compiled rules.
- Readers need a consistent view across several fields and must not block.
- The state is small enough that rebuilding it is cheap relative to how often it changes.
Trade-offs
- Every write copies the whole structure, so the cost scales with size rather than with the size of the change.
- Readers may act on a snapshot that is already stale — which is usually fine, and must be stated rather than assumed.
- Two readers in the same request can see different snapshots unless one is captured and passed down.
- Allocation churn moves the cost to the allocator and the garbage collector.
How it fails
- Unsafe publication: the reference is visible before the object's fields are, and a reader sees a half-built structure. This is the double-checked locking bug.
- Deep-immutability violation — the outer object is immutable but holds a mutable list, so the guarantee is fictional.
- Read-modify-write on the reference (read, build a new version from it, store) loses concurrent updates unless it is a compare-and-swap loop.
- Frequent writes turning the copy cost into the dominant cost, at which point a lock would have been cheaper.
- Staleness assumptions that are correct for config and wrong for the balance someone spends against.