USE: Utilization, Saturation, Errors
For every resource, ask three questions: how busy is it, is work waiting for it, and is it failing? The middle question is the one that matters and the one most dashboards omit — which is why "CPU is only 40%" keeps getting offered as evidence that CPU is fine.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Utilization is not saturation
Utilization is the fraction of time a resource is busy. Saturation is the amount of work that could not be served immediately and had to wait. Errors is work the resource refused outright. The three are independent, and the confusion between the first two is probably the single most common analytical error in production performance work.
The reason they diverge is queueing. A resource at 40% average utilization can still have a deep wait queue if arrivals are bursty — utilization is an average over a window, and a window that includes idle seconds and saturated milliseconds reports a comfortable middle. Conversely a batch worker pinned at 98% with no queue is a resource being used efficiently, exactly as intended (see Queueing: Why Systems Get Slow Before They Get Broken for why waiting time rises non-linearly near capacity).
This is why the actionable question is never "how busy is it". It is "is anything waiting for it, and for how long". Wait time is the signal that translates directly into user-visible latency; utilization is a capacity-planning input (see Headroom: The Capacity You Deliberately Do Not Use) that becomes an incident signal only in combination with saturation.
| Utilization | Saturation | Reading | Right response |
|---|---|---|---|
| Low | Low | Genuinely idle. The constraint is elsewhere. | Look at another resource or a dependency |
| Low | High | Bursty arrivals, or a limit far below capacity — a pool sized at 20 on a 32-core box | Raise the limit or smooth arrivals; do not add hardware |
| High | Low | Efficient use. This is what a well-tuned batch worker looks like. | Nothing — leave it alone |
| High | High | The resource is the constraint and is over its comfortable capacity. | Add capacity, shed load, or make each unit of work cheaper |
The three questions, per resource
USE is applied resource by resource, and the value comes from being exhaustive: enumerate every resource the request path contends on, then get all three readings for each. The enumeration is the part teams skip, which is how a connection pool goes unwatched for two years.
Note how different the saturation signal is per resource. For CPU it is run-queue length or scheduler wait, not utilization percentage. For memory it is swap activity, page-fault rate or allocation stalls, not resident size. For a connection pool it is the count of waiters and their wait time. For a disk it is the I/O queue depth. Each of these needs to be instrumented specifically; none is derivable from the utilization number that dashboards default to showing.
Errors, the third question, is the one that turns a slow system into a failing one, and it usually appears last. A pool that runs out of connections first makes requests wait, then — once waits exceed the acquisition timeout — starts refusing them. That transition from saturation to errors is the moment latency degradation becomes an outage, and watching saturation is what gives you warning before it happens.
| Resource | Utilization | Saturation (the signal that matters) | Errors |
|---|---|---|---|
| CPU | Busy time per core | Run-queue length; scheduler wait time; cgroup throttling | Throttled periods, OOM-killed processes |
| Memory | Resident bytes in use | Page-fault rate, swap activity, allocation stalls, GC pause time | OOM kills, allocation failures |
| Disk | Device busy percentage | I/O queue depth, await time | I/O errors, write failures |
| Network | Bandwidth used vs link capacity | Send/receive buffer occupancy, retransmits | Drops, connection resets |
| Connection pool | Connections in use / pool size | Waiters count and wait time | Acquisition timeouts |
| Worker / thread pool | Busy workers / total workers | Queued tasks and their age | Rejected submissions |
| Queue / broker | Consumer throughput vs capacity | Depth and oldest-message age | Dead-lettered messages |
The reading that ends the "we need bigger instances" argument
The most common shape of this mistake: latency is bad, CPU sits at 22%, memory is flat, and someone proposes scaling up because "the servers must be struggling". USE applied properly ends that conversation in about ninety seconds, because it forces the question "which resource, and what is waiting for it" rather than "does anything look busy".
In the reading below, three resources are comfortable and one is completely saturated. Adding instances would help only incidentally — it would add outbound pool capacity as a side effect of adding processes, at far higher cost than raising the pool limit. Worse, if the constraint had been the *provider's* concurrency limit rather than the local pool, more instances would have made the problem worse by increasing concurrent pressure on a dependency already at its limit.
This is also where USE and RED meet. RED said customers are waiting (see RED: Rate, Errors, Duration); USE says what they are waiting *for*. Neither alone would have prevented the scale-up proposal; together they name the fix precisely.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| CPU · utilization | 22% | Not busy. On its own this is the number that gets misread as "healthy" | normal |
| CPU · run-queue length | 0.3 | Nothing waiting for a core — CPU is genuinely not the constraint | normal |
| Memory · resident | 61% of limit, flat | Stable, no growth trend | normal |
| Memory · GC pause time | 4ms/s | Negligible pause contribution to latency | normal |
| DB pool · in use | 18 / 20 | High utilization — suspicious but not conclusive on its own | suspect |
| DB pool · waiters | 2, mean wait 8ms | Slight queueing, small contribution to request latency | suspect |
| Outbound HTTP pool · in use | 32 / 32 | Fully utilized | suspect |
| Outbound HTTP pool · waiters | 84, mean wait 1,240ms | The constraint: 1.24s of the request is spent waiting for a connection slot | smoking gun |
Key points
- Utilization is how busy; saturation is what is waiting. They diverge under bursty arrivals and under limits set below capacity.
- Low utilization with high saturation is the combination that gets misdiagnosed as "no resource problem".
- The saturation signal is different per resource — run-queue for CPU, waiters for a pool, queue depth for a disk — and is rarely instrumented by default.
- Errors are the late signal: a saturated pool first makes requests wait, then starts refusing them. Saturation is your warning.
- RED says customers are waiting; USE says what they are waiting for. Use both.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Arrivals → resource: concurrent demand exceeds the resource's immediate capacity, either because demand rose or because each unit of work now takes longer.
- 2Resource → queue: excess work waits rather than executing, and utilization stays moderate because the resource is fully occupied only in bursts.
- 3Queue → latency: waiting time is added to every affected request, and it lands on the tail before it lands on the median.
- 4Latency → occupancy: slower requests hold their resource longer, so occupancy climbs — the self-reinforcing loop that turns queueing into saturation.
- 5Saturation → errors: waits exceed the acquisition timeout, requests start being rejected, and clients retry into an already-saturated resource (see Retry Storms: The Load You Generated Yourself).
- • "CPU is at 40%, so CPU is fine." Possibly, but only the run-queue reading establishes that. Averaged utilization hides bursts, and container CPU throttling can be severe at moderate average utilization.
- • "The pool is at 100%, that is the problem." Only if there are waiters. A pool that is fully used with an empty wait queue is correctly sized.
- • "We need bigger instances." Bigger instances change CPU and memory. If the constraint is a pool limit, a dependency's concurrency cap or a lock, more hardware changes nothing — and can make dependency pressure worse.
- • "Memory is at 80%, we are about to OOM." Resident memory including caches is not the same as memory pressure; the fault rate and allocation stalls are the signals (see Reading Memory: RSS, Heap, Working Set and the Number on Your Dashboard).
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Enumerate every bounded resource in the request path first: CPU, memory, disk, network, and every pool, queue and semaphore.
- • For each, get all three: utilization, the resource-appropriate saturation signal, and the error/rejection count.
- • Prefer wait *time* over queue *length* where available — wait time converts directly into user-visible latency.
- • Read saturation at a high enough resolution to see bursts; a one-minute average hides sub-second queueing entirely.
- • Instrument saturation for every pool and queue in the request path — waiters, wait time, rejections — not only utilization.
- • Where saturation is high and utilization is low, raise the limit or smooth arrivals rather than adding hardware.
- • Where both are high, choose deliberately between adding capacity, shedding load and making each unit of work cheaper (see [[concurrency-limits]]).
- • Set acquisition timeouts and surface rejections explicitly, so the saturation-to-errors transition is visible rather than silent.
- • Add the resource enumeration to the service dashboard so the next investigation does not have to rediscover which pools exist.
- • After the change, waiters and wait time on the identified resource should drop to near zero at comparable load.
- • The corresponding share of request latency should shrink in the trace breakdown — confirm in a waterfall, not only in the resource metric.
- • Watch for the constraint moving: after relieving one resource, re-run the USE pass to find the new one (see [[bottleneck-migration]]).
- • Saturation instrumentation is per-resource work — every pool library exposes different counters, and some expose none.
- • High-resolution saturation data is more expensive to store than one-minute utilization averages.
- • Raising limits to relieve local saturation can push pressure onto a shared downstream dependency, converting a local problem into a global one.
- • Alert on saturation signals (waiters, wait time, queue age) rather than on utilization thresholds, which page too often and too late.
- • Add a load test that drives the identified resource to saturation, so a future limit change is caught before production.
- • Record pool and limit values in configuration review; silent limit changes are a common cause of recurrence.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEAll readings are a constructed teaching example chosen to show the low-utilization/high-saturation shape. Real values depend entirely on workload and configuration.
- ENVIRONMENT-SPECIFICSaturation signals differ by platform: cgroup throttling on containers, scheduler wait on bare metal, and credit exhaustion on burstable cloud instance types are all distinct mechanisms.