Computing or Waiting?
The first fork in every performance investigation. A CPU-bound service wants better algorithms or more cores; an I/O-bound service wants concurrency, batching or a faster dependency. Applying either fix to the other problem reliably makes things worse.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Telling them apart from the outside
The distinction is about where the wall-clock time goes. CPU-bound: the process has work to do and is limited by how fast it can execute instructions. Utilization sits near its limit, the run queue grows because threads are waiting for a core, and voluntary context switches are low because the process rarely gives up the CPU willingly. I/O-bound: the process is blocked on something external. Utilization is low, voluntary context switches are high (each block is a yield), and the threads are parked in read, epoll_wait or a lock.
The trap is that both produce identical user-visible symptoms — high latency, poor throughput, unhappy customers — and both get worse under load. Guessing produces confident wrong action: adding cores to an I/O-bound service changes nothing except the bill, and adding concurrency to a CPU-bound service makes it worse by adding context-switch overhead and lengthening the run queue.
One clarification worth making: "CPU-bound" is a property of the process at a moment, not of the service forever. The same service can be CPU-bound during a cache-miss storm and I/O-bound the rest of the time (Cache Stampede: Everyone Misses at Once), which is why the measurement must come from the incident window rather than from general knowledge about the service.
| Signal | CPU-bound | I/O-bound |
|---|---|---|
| CPU utilization vs limit | Near saturation (>85% of quota) | Low (often <30%) despite high latency |
| Run queue length | Grows with load — threads waiting for a core | Near zero — nothing wants the CPU |
| Voluntary context switches | Low: the process rarely yields | High: every block is a yield |
| Thread states | Runnable / running | Blocked in read, epoll, futex |
| CPU profile | Full: real frames with real self time | Nearly empty, or dominated by wait frames |
| Adding concurrency | Makes it worse (switching overhead) | Helps until the dependency saturates |
| Adding cores | Helps roughly linearly, up to contention | Changes nothing |
The fixes do not transfer
For a CPU-bound service the levers are: do less work per request (better algorithm, fewer allocations, less serialization — Algorithmic Cost in a Request Handler and Allocation Rate Is a Cost Even Without a Leak), do the work elsewhere (cache the result, precompute, offload), or buy more CPU (more cores or more instances). Concurrency is not a lever; the cores are already busy, and more threads mean more context switching for the same throughput.
For an I/O-bound service the levers are: wait for fewer things (batch, cache, remove a dependency from the critical path), wait concurrently instead of serially (Sequential or Parallel: Same Work, Different Latency), or make the dependency faster. Here concurrency *is* the primary lever — a service waiting 80% of the time can handle far more in-flight requests with the same CPU, which is the entire argument for async I/O. More cores are close to useless.
The mixed case is common and worth naming: a service that is I/O-bound at low load can become CPU-bound at high load once concurrency rises enough to keep the cores busy — or once GC starts consuming CPU, which is the Allocation Rate Is a Cost Even Without a Leak pathway. The measurement has to be repeated at the load level you care about, and "we profiled it in staging at 5% of production traffic" is not that measurement.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| A: cpu utilization | 96% of 2-core quota | Saturated — this one is computing | smoking gun |
| A: run queue | 5.2 | Threads queued for a core; latency is scheduler wait | smoking gun |
| A: voluntary ctx switches | 180/s | Low — rarely yields, consistent with CPU-bound | normal |
| B: cpu utilization | 11% of 2-core quota | Idle while p99 is 4 s — it is waiting for something | smoking gun |
| B: run queue | 0.1 | Nothing wants the CPU | normal |
| B: voluntary ctx switches | 41,000/s | Constantly blocking and yielding — classic I/O-bound | smoking gun |
The cases that fool the test
Three situations break the simple reading. Container CPU quotas: a process at "40% CPU" measured against the host may be at 100% of its cgroup quota and getting throttled hard. Throttled time is the signal that matters, and it is invisible if you read host-level utilization — one of the most common misdiagnoses in containerized environments (CPU Saturation: When Cores Become the Queue).
Lock contention looks I/O-bound by every external signal — low CPU, high voluntary context switches, blocked threads — but the blocking is on other threads in the same process, not on external I/O. The fix is neither more cores nor more concurrency; it is reducing the critical section (Low CPU, High Latency: Lock Contention). An off-CPU or lock profile distinguishes it immediately.
Single-threaded runtimes distort the utilization reading. A Node.js process pinned at 100% of one core on an 8-core machine reads as 12.5% host utilization while being completely saturated (Event-Loop Lag: One Callback, Everybody Waits). Always compare against the limit that actually applies — the quota, the core the process can use, the event loop's capacity — rather than against total machine capacity.
Key points
- CPU-bound shows near-saturated utilization, a growing run queue and low voluntary context switches; I/O-bound shows low CPU, an empty run queue and constant yielding.
- The fixes are not interchangeable: concurrency helps I/O-bound and hurts CPU-bound; more cores help CPU-bound and do nothing for I/O-bound.
- Measure utilization against the limit that applies — cgroup quota, single-core capacity for single-threaded runtimes — not against host capacity.
- Lock contention mimics I/O-bound on every external signal; only an off-CPU or lock profile separates them, and its fix is different again.
- The classification is a property of the process under a specific load, not a permanent label — re-measure in the window that matters.
Progressive depth
Overview
Two reasons a program is slow: it has too much to do, or it is waiting for someone else. Check whether the CPU is busy. Busy means the first; idle means the second. The fixes are completely different.
Practical
Read CPU utilization against the applicable limit, run-queue length and the voluntary/involuntary context-switch split. Near-saturated plus a growing run queue is CPU-bound; low CPU plus tens of thousands of voluntary switches is blocking.
Advanced
Watch for the three fooling cases: cgroup throttling (host CPU lies), lock contention (looks I/O-bound, needs a different fix), and single-threaded runtimes (100% of one core reads as 12% of the host). An off-CPU profile resolves all three.
Internals
The distinction is really about thread state in the scheduler. A CPU-bound thread cycles runnable → running and is preempted involuntarily when its quantum expires. A blocking thread calls into the kernel, is moved to a wait queue, and yields voluntarily — which is why the context-switch split is diagnostic. See The Scheduling Problem and Context Switching for the mechanism.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Latency → engineer: p99 at 4 s, which is compatible with either diagnosis and therefore decides nothing.
- 2CPU utilization → engineer: 11% against quota, ruling out compute as the constraint.
- 3Context switches → engineer: 41,000 voluntary switches per second, meaning threads block and yield constantly.
- 4Off-CPU profile → engineer: threads parked in socket reads against the payments dependency, not in futexes.
- 5Dependency latency → engineer: the provider's p99 accounts for nearly all of the request time (What Changes When Work Crosses a Machine).
- • "CPU is at 40%, so we have headroom." Against the host, maybe. Against a 0.5-core cgroup quota, that process is saturated and being throttled.
- • "CPU is low, so add threads." If the blocking is lock contention, more threads increase contention and make it worse.
- • "The CPU profile is empty, the profiler is broken." An empty CPU profile in a slow process is a definitive finding: it is not computing.
- • "It was I/O-bound last month." Cache hit rates, data volumes and traffic all change the answer. Re-measure.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Read CPU utilization against the process's actual limit, plus cgroup throttled time if containerized.
- • Check run-queue length and the voluntary/involuntary context-switch split — together they separate the two cases faster than any profile.
- • Take a CPU profile: a nearly empty one is itself the finding, and redirects you to off-CPU or lock profiling.
- • Sample thread states during the slow window to see where threads actually are (runnable, blocked on socket, blocked on futex).
- • CPU-bound: reduce work per request first (algorithm, allocations, serialization), then add cores or instances — in that order, because scaling an inefficiency is expensive forever.
- • I/O-bound: raise concurrency so waiting overlaps, and remove or batch dependencies on the critical path.
- • Quota-bound: raise the limit or reduce per-request CPU; throttling is a configuration problem masquerading as a code problem.
- • Lock-bound: shrink the critical section or shard the lock — neither cores nor concurrency will help ([[lock-contention]]).
- • CPU-bound fixes should show CPU seconds per request falling and throughput per instance rising at unchanged latency.
- • I/O-bound fixes should show concurrency rising with CPU roughly unchanged, and latency falling because waits now overlap.
- • Confirm the constraint actually moved by re-running the same signal set — a fix that leaves the classification unchanged did not address the constraint ([[bottleneck-migration]]).
- • Watch the dependency's own metrics after raising concurrency: you may have relocated saturation downstream ([[saturation]]).
- • Raising concurrency on an I/O-bound service pushes load onto the dependency and can convert your latency problem into its saturation problem.
- • Adding cores to a CPU-bound service is fast and permanent-feeling, and it hides an efficiency problem that returns at the next traffic step.
- • Off-CPU profiling has higher overhead than CPU profiling in some runtimes, so it is not always suitable for always-on use.
- • Reducing per-request CPU usually costs code complexity or readability, which is a real maintenance cost.
- • Dashboard CPU utilization against quota (not host) alongside throttled time, so the containerized misreading cannot recur.
- • Alert on CPU seconds per request and on concurrency, so both classes of regression are visible.
- • Include the classification and its evidence in the runbook for the service — it saves the first fifteen minutes of every future incident.
- • Re-measure after significant traffic or data-volume growth, since the classification can flip without any code change.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe two service dashboards are constructed to contrast cleanly; real systems are frequently mixed and shift between states under load.
- ENVIRONMENT-SPECIFICUtilization only means something relative to the applicable limit — cgroup quota, single-core capacity, or hyperthread contention all change what "80% CPU" implies.