The four meanings of 100% CPU

At least five different things: (1) legitimately busy — throughput-bound work, correlated with load; (2) a hot or infinite loop in user code — a retry without backoff, a spin on a condition; (3) lock contention or busy-waiting — threads spinning or bouncing through futex; (4) a syscall storm — tiny reads and writes, stats, epoll_wait with zero timeout; (5) garbage collection or a runtime pathology — a heap near its limit collecting continuously. Each has a different fix, so the job is to classify before profiling (The OS Debugging Playbook: Four Symptoms, Fourteen Causes).

First, top: the us/sy split. High user time points at the application (1, 2, 5); high system time points at the kernel — syscalls or contention on kernel locks (3, 4). wa high means the process is waiting, not computing, whatever the load average says. Then top -H -p <pid> for per-thread usage: one thread at 100% and the rest idle is a loop or a single-threaded bottleneck; all threads at 100% with throughput *lower* than before is contention; N threads named GC at 100% is the collector.

Second, strace -c -p <pid> for ten seconds (or perf trace): a hot loop makes no syscalls; a syscall storm shows millions of read/write/stat; lock contention shows futex dominating; a busy event loop shows epoll_wait returning constantly. Third, two stack samples ten seconds apart — gdb -p + bt, jstack, py-spy dump: the same frame twice is a loop or a spin; different frames is real work. Only then a profiler or a flame graph to name the function (System Calls).

Remember what "100%" means: top reports per-process CPU as a percentage of one core, so 100% on a 16-core box is one core — a single-threaded bottleneck, not saturation. And a process can be *slow* without being at 100%: cgroup throttling, steal time on an oversubscribed VM (st in top), or frequency scaling look like slowness with spare CPU.