The question this answers
Is this unit of work spending its wall clock on a core or in a wait queue, and what does that dictate about how to run it?
One "generate export" handler: fetch 200 000 rows from Postgres (waiting), transform them to CSV (computing), gzip the result (computing), upload to object storage (waiting), write an audit row (waiting).
The row buffer as it passes between phases, the connection pool used by phases 1 and 5, and — if the transform is parallelised — the output ordering. Within a phase, nothing else.
Every fetched row appears exactly once in the uploaded CSV, in query order, and the audit row is written if and only if the upload succeeded — whatever model each phase runs under.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
The two ends of the axis, and the honest middle
Performance & Observability owns diagnosing this from live signals — [[cpu-bound-vs-io-bound]] is the lesson that teaches you to read the graphs. This lesson takes the classification as an input and asks the design question: given that answer, what execution model does this work get?
The mapping is unglamorously direct. Waiting-bound work wants a model where a waiting unit releases its execution context: async, coroutines, an event loop, or simply more threads if the counts are small. Compute-bound work wants a model that can occupy more cores: threads in a runtime that permits it, processes in one that does not, or a data-parallel primitive.
The trap is the third row of the matrix. Almost nothing real is purely one or the other, and the mistake is classifying the *handler* rather than its *phases*. This export handler is 88% waiting by wall clock, so a superficial reading says "async, done" — and then gzip on the event loop stalls every other request on the process for 900 ms. Phases get classified, not handlers.
| Class | Examples | What the wall clock is spent on | Model that fits | Failure when you pick the other one |
|---|---|---|---|---|
| CPU-bound | gzip, image resize, AES, JSON/CSV parsing, matrix maths, template rendering, PDF layout, hashing | a core retiring instructions | parallelism: threads on distinct cores, a small pool, processes where threads cannot, or SIMD | On an event loop it serialises AND blocks every neighbour. Async adds suspension points and returns nothing. |
| I/O-bound | HTTP calls, SQL queries, disk reads, object storage, third-party APIs, queue consumption, DNS | a wait queue, with the core free | concurrency: async/await, coroutines, an event loop, or a bounded thread pool | Cores are useless here — the CPU was already idle. Thread-per-wait works but costs stacks and switches for nothing. |
| Mixed (most real handlers) | fetch → transform → upload; receive → decode → persist; query → render → return | both, in distinct phases | both, split: loop or async for the waiting phases, a bounded worker pool for the computing phase, a queue between | Classifying the handler by its dominant phase puts CPU work on the I/O model. The 12% of the time that computes stalls the 88% that does not. |
| Memory-bandwidth-bound | large array scans, big joins, streaming reductions over multi-GB data | waiting on memory while the core is nominally "busy" | fewer bytes per operation before more cores — layout, smaller types, fused passes | Looks CPU-bound on every graph. Adding cores contends for the resource that was already saturated. |
The classification procedure
The single number that decides it is CPU time over wall-clock time for the phase. If a phase takes 900 ms of wall clock and 890 ms of CPU, it computes. If it takes 500 ms of wall clock and 2 ms of CPU, it waits. There is no ambiguity in either case and the measurement is cheap in every runtime.
The follow-up question — the one that separates "add cores" from "you cannot add cores" — is whether the compute-bound phase is partitionable. gzip over one stream is not: it is inherently sequential over its window. gzip over 200 independent chunks is. Compression that cannot be split is a phase you move off the loop but cannot make faster with more workers, and knowing that before you build a pool saves a week.
The last branch matters for cost rather than latency: if a phase is compute-bound *and* not on the critical path of a user-visible request, the right answer is usually neither concurrency nor parallelism but relocation — enqueue it, answer the user, and let a worker fleet do it. See [[background-jobs]].
The export handler, phase by phase
Laid out over time, the handler is obviously two different animals wearing one function. The fetch, upload and audit phases hold no core at all. The transform and gzip phases hold one core solidly and hold it for 1.4 seconds combined.
On a single-threaded event loop, that 1.4 seconds is not "12% of a slow handler". It is 1.4 seconds during which every other request on the process makes zero progress, including the health check. Ten concurrent exports on one loop is fourteen seconds of total stall spread across every unrelated user of that process. This is the failure that [[blocking-the-event-loop]] is named after, and phase classification is what prevents it.
The fix follows directly from the picture: keep phases 1, 4 and 5 on the loop where their waiting overlaps with everything else, and move phases 2 and 3 to a bounded worker pool. The queue between the two is not overhead — it is the place you get to see and limit how much CPU work is outstanding, which is a property the naive version does not have at any price.
Key points
- Classify phases, not handlers. A handler that is 88% waiting can still hold a core for 1.4 seconds and stall everything around it.
- CPU time divided by wall time is the measurement. Below roughly 0.2 is waiting; above roughly 0.8 is computing.
- Waiting-bound work wants concurrency. Compute-bound work wants parallelism. Neither substitutes for the other.
- Compute-bound is not automatically parallelisable: single-stream compression is compute-bound and inherently sequential.
- Memory-bandwidth-bound work looks CPU-bound on every graph and does not respond to more cores. Check achieved bandwidth before adding workers.
- For compute-bound work that is not on a user-visible path, relocation beats both models: enqueue it and answer immediately.
- The queue between an I/O phase and a CPU phase is where you get visibility and a limit — that is a feature, not overhead.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Break the unit of work into phases at every boundary where it starts or stops waiting on something external.
- • For each phase, measure CPU time and wall-clock time over the same interval and take the ratio.
- • Classify: below ~0.2 waiting-bound, above ~0.8 compute-bound, in between means the phase is still two phases.
- • For compute-bound phases, ask whether the input partitions into independent chunks; if not, parallelism is unavailable and offloading is the only move.
- • For compute-bound phases, check achieved memory bandwidth against the machine's peak before assuming more cores will help.
- • Assign a model per phase and put an explicit, bounded hand-off between phases with different models.
- • Ten exports arrive on one loop: their fetch phases overlap perfectly, then their gzip phases run strictly one after another, and total latency is roughly ten times gzip regardless of how much concurrency the fetch enjoyed.
- • Export A is mid-gzip when B's fetch completes; B is ready but not running for 900 ms. B's trace attributes that 900 ms to nothing at all, because no span was open for it.
- • Transform parallelised across four workers: chunks complete in the order 2, 0, 3, 1, and if the CSV is written in completion order the export contains every row exactly once in the wrong order — the invariant's "in query order" clause is what catches it.
- • The gzip worker pool has two slots and ten exports arrive: eight queue. The queue is bounded and visible, which is the entire reason to have created it, and the alternative was ten stalls on the loop.
- • Classification tells you which model can possibly help. It does not tell you that it will: a partitionable compute phase can still be bandwidth-bound and gain nothing.
- • Moving CPU work off the event loop guarantees the loop stays responsive. It guarantees nothing about the CPU work getting faster — one worker is still one core.
- • A bounded hand-off queue guarantees the amount of outstanding CPU work is knowable and capped. It does not guarantee callers get served; past the cap they wait or are rejected, and that must be a decision. See
[[concurrent-backpressure]]. - • The CPU-over-wall ratio is a property of the phase on this hardware with this data, not a universal label. A phase that waits on a warm cache computes on a cold one.
- • Waiting phases contend for the connection pool and for whatever the upstream limits — not for cores.
- • Computing phases contend for cores, for last-level cache and for memory bandwidth, and those three are hard to distinguish from application-level metrics.
- • A mixed handler on one loop makes the computing phase contend with every unrelated waiting phase, which is the pathological case: work that needs no core is blocked by work that needs one.
- • The hand-off queue is contention made explicit and measurable, which is why introducing it usually improves diagnosis even before it improves latency.
- • Event-loop starvation from a CPU phase inside a nominally I/O-bound handler — the single most common concurrency incident in server-side JavaScript and Python.
- • Thread pool sized for I/O (say 200) used for CPU work: 200 runnable threads on 8 cores, and throughput falls while CPU rises. See
[[oversubscription]]. - • Thread pool sized for CPU (say 8) used for I/O work: 8 concurrent requests and a 500-deep queue, with the box 96% idle.
- • Order loss when a compute phase is parallelised and results are collected in completion order.
- • Misclassification via averages: a handler whose mean says "I/O-bound" while p99 is dominated by a compute path that only large inputs take.
- • Before choosing any library. The classification is fifteen minutes of measurement and it eliminates entire branches of the design space.
- • When a service has been "optimised" twice with no result — usually the model matches the dominant phase and the incident comes from the other one.
- • When sizing pools: the class of work is the only principled input to the sizing conversation, and it still does not give you a formula. See
[[thread-pool-sizing]].
- • When the phases are so short that splitting them costs more than they take. A 2 ms handler does not need a phase model.
- • When it becomes an excuse to build a pipeline: two phases, two pools and a queue is real operational surface, and for a low-traffic endpoint the sequential version is better engineering.
- • When classification is done once and treated as permanent. Adding a cache turns a waiting phase into a computing one, and the model quietly stops matching.
- • Per-phase CPU time versus wall-clock time —
process.cpuUsage()deltas,resource.getrusage, or CPU-seconds from the profiler over the span. - • Event-loop lag or scheduler run-queue delay during the handler, which converts "the loop was blocked" from a theory into a number.
- • Per-core utilisation during the compute phase: one core pinned and the rest idle means parallelism is available and unused.
- • Achieved memory bandwidth against peak for large-array phases — this is the check that stops you buying cores you cannot use.
- • The p50/p99 split by input size. Classification derived from the mean misses the compute-heavy tail entirely.
- • Splitting a handler into phases with different models turns one function into a small pipeline with a queue, a pool and two failure surfaces.
- • The hand-off must serialise data across a worker boundary, and for large payloads the copy can consume the gain.
- • Cancellation and timeouts have to cross the boundary too, or a cancelled request leaves a worker computing a result nobody wants. See
[[cancellation-propagation]]. - • Two pools mean two capacity decisions and two saturation modes, and the interaction between them is not obvious from either one.
- • Make the phase disappear. The cheapest export is one that streams rows to storage and never materialises a 200 000-row buffer to compress.
- • Move the compute to where the data is:
COPY ... TO PROGRAM 'gzip'or a database-side aggregate avoids both the transfer and the phase. - • Use a library that already crosses the boundary correctly — a streaming gzip that yields between chunks keeps the loop alive with no pool at all.
- • Relocate the entire handler to a job queue and give the user a status endpoint. See
[[async-job-pattern]].
Concurrency lab
They come from a queueing and contention model inside Engineer Atlas. What is faithful is the behaviour: work that waits benefits from more workers, work that computes does not, a wide critical section pins parallelism near 1 no matter how many cores you buy, and arrivals past capacity produce an unbounded queue rather than a large latency. Real arrivals are burstier than this model assumes, so real systems reach every one of these walls earlier than the sliders suggest. Do not quote a millisecond from this page.
CPU parallelism simulator
Amdahl’s term, a synchronisation term, an oversubscription term and a bandwidth ceiling, each one a knob you can switch off. Real curves have more causes than four and are rarely this smooth. There is no ideal core count to read off this chart.
Three I/O calls, one thread
sequential const a = await getProfile(); const b = await getFlags(); const c = await getOrders() concurrent const [a, b, c] = await Promise.all([getProfile(), getFlags(), getOrders()]) wall clock 261 ms → 127 ms (2.06× less waiting) CPU used 21 ms → 21 ms (identical — no extra core was touched)
What people believe, and what is true
This handler is I/O-bound, so async is enough.
The handler is 88% waiting and still holds a core for 1.4 seconds. The 12% is what stalls the loop, and the dominant phase tells you nothing about it.
CPU-bound means it will parallelise.
Single-stream gzip is entirely CPU-bound and entirely sequential. Compute-bound says "not concurrency"; only partitionability says "parallelism".
The box is at 100% CPU, so we need more cores.
It may be memory-bandwidth-bound, in which case the cores are stalled on loads and look busy. Check achieved bandwidth before buying anything.
Go deeper
Overview
Ask what the work does with its wall clock: sit on a core, or sit in a queue. Computing wants cores; waiting wants overlap.
Practical
Measure CPU time over wall time per phase. Below 0.2 is waiting, above 0.8 is computing, in between means you have not split finely enough yet.
Advanced
The classification is a property of a phase on specific hardware with specific data, and it moves. Adding a cache converts waiting into computing; growing the dataset converts computing into memory-bandwidth-bound. Re-derive it when the system changes rather than inheriting a label.