The question this answers
The service is stalled and CPU is near zero. What does a snapshot of every thread tell me?
A stalled 24-thread request pool in a service that has stopped answering, taken as a single point-in-time capture of every thread's state and call stack.
Whatever the stacks show them all sitting on: in the canonical case, one lock, one connection pool, or one downstream socket.
Every worker thread is either executing a request, waiting for a resource it can name, or idle in the pool — no thread is in a state the dump cannot account for.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
Four states, and what each one means about the system
The vocabulary varies by runtime but the categories do not. Running means on a CPU right now. Runnable means ready and waiting for a core — many runnable threads with few cores is oversubscription, not blockage. Blocked means waiting to acquire a lock another thread holds. Waiting means parked on a condition, a future, a queue or an I/O completion, with no owner to point at. Sleeping is waiting with a timer attached.
The distinction that carries the diagnosis is blocked versus waiting. Blocked has a *culprit*: some thread owns the thing. Waiting usually does not — the thread is idle by design, and a pool of workers parked on an empty queue is a healthy system, not a stalled one. Confusing the two is why people report "all threads are waiting!" about a service that is simply not busy. The kernel-side model is Process States.
Read counts before you read stacks. Twenty-four threads, of which one is running and twenty-three are blocked, is a completely different system from twenty-four threads all waiting on a socket read, and you can tell which before looking at a single frame.
| State | Means | Has a culprit? | First question |
|---|---|---|---|
| RUNNING | On a CPU executing instructions. | No | Is the stack in application code or in the runtime (GC, JIT, allocator)? |
| RUNNABLE | Ready to run, waiting for a core. | No — the scheduler | How many runnable versus how many cores? Many means oversubscription. |
| BLOCKED | Waiting to enter a monitor / acquire a lock. | Yes — whoever owns it | Which lock, and what is the owning thread doing right now? |
| WAITING | Parked on a condition, future, queue or I/O with no timeout. | Usually not | Is this idle-by-design (empty pool queue) or a lost wakeup? |
| TIMED_WAITING / SLEEPING | Parked with a deadline. | Usually not | Is the timeout long enough to look like a hang? A 60s socket timeout stalls for 60s. |
The dump, and the one line that solves it
The conceptual dump below is the classic shape. Twenty-two threads are blocked on the same lock object. One thread owns it. That one thread is not computing — it is inside an HTTP read, waiting for a downstream service, *while holding the lock*. That is the whole incident, and it is visible in about four seconds of reading if you know to look for the owner rather than the crowd.
The crowd is not the bug. Twenty-two identical stacks are twenty-two symptoms of one cause, and treating them as twenty-two problems — "why are so many threads blocked?" — leads people to add threads, which adds waiters. The question a dump answers is always *who holds it, and what are they doing*.
Two other shapes are worth recognizing on sight. If no thread owns the lock and everyone is waiting on a condition, suspect a lost wakeup — Lost Wakeups: The Notify That Arrived Before the Wait. If thread A is blocked on lock L1 owned by B, and B is blocked on L2 owned by A, you are looking at a deadlock cycle, and the dump has just handed you the wait-for graph for free — Deadlock and Lock Ordering.
THREAD DUMP -- pid 4412 -- 24 pool threads, 3 runtime threads
summary: RUNNING 1 RUNNABLE 0 BLOCKED 22 WAITING 1 TIMED_WAITING 3
"pool-worker-7" state=RUNNING holds: lock@0x7f3a <-- THE OWNER
at socket.read(...)
at http.client.awaitResponse(...)
at inventory.fetchStock(...) <-- I/O inside the critical section
at catalog.refreshIndex(...) [synchronized on lock@0x7f3a]
at handler.getProduct(...)
"pool-worker-3" state=BLOCKED waiting to lock: lock@0x7f3a owner=pool-worker-7
at catalog.refreshIndex(...)
at handler.getProduct(...)
"pool-worker-4" state=BLOCKED waiting to lock: lock@0x7f3a owner=pool-worker-7
at catalog.refreshIndex(...)
at handler.getProduct(...)
... 20 further threads, byte-identical stacks, same lock, same owner ...
"pool-worker-22" state=WAITING on: taskQueue.notEmpty
at queue.take(...)
at pool.workerLoop(...) <-- idle by design, NOT a symptom
"scheduler" state=TIMED_WAITING on: parkNanos(30s)
"gc-thread-1" state=RUNNING
"metrics-scrape" state=BLOCKED waiting to lock: lock@0x7f3a owner=pool-worker-7
<-- even the metrics exporter is stuckOne dump is a photograph; two are a story
A single dump cannot distinguish "stuck" from "busy". A thread found in parse() might be permanently wedged or might have been there for eight microseconds. The fix is to take three dumps thirty seconds apart. Frames that are identical across all three are genuinely stuck; frames that move are working. That comparison converts a snapshot into evidence, and it is the single most useful habit in this lesson.
The corollary is that a dump is a *sample*, with all the honesty problems of one. It is biased toward long operations by construction — you are far more likely to catch a thread inside a 300ms call than inside a 3µs one — which makes it excellent for finding stalls and useless for finding a hot loop. That is what a CPU profile is for; see Off-CPU Time: The Thing a CPU Profiler Cannot See and Reading a Flame Graph.
Dumps also have a cost. Many runtimes must reach a safepoint to walk stacks, which briefly stops the world; capturing a dump of a thousand threads during an incident can add a visible pause. That is usually worth it, and it is a reason not to put dumps on a one-second timer.
Key points
- Read the state summary before any stack: one running and twenty-two blocked is a different system from twenty-four waiting on sockets.
- BLOCKED has an owner; WAITING usually does not. That distinction is the difference between a culprit and an idle pool.
- Twenty threads in one lock frame is one problem, not twenty. Find the owner and read what the owner is doing.
- The classic finding is I/O inside a critical section — the owner is not computing, it is waiting on a network, and everyone is behind it.
- One dump is a photograph. Take three, thirty seconds apart: frames that repeat are stuck, frames that move are fine.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Signal the runtime to capture stacks for every thread; most runtimes require reaching a safepoint first, which briefly halts execution.
- • For each thread record: name, state, the resource it waits on, and — critically — the identity of the owner when the state is BLOCKED.
- • Aggregate identical stacks so twenty-two copies collapse to one entry with a count; the count is the fan-in, the stack is the symptom.
- • Follow every blocked thread's wait target to its owner, and read the owner's stack. That is where the cause is.
- • Repeat two more times at a fixed interval and diff, so "stuck" is demonstrated rather than assumed.
- • worker-7 acquires the index lock; worker-7 issues a downstream HTTP call inside the section; workers 3..24 arrive and block; the metrics exporter arrives and blocks; the service now cannot even report its own state. No invariant is violated — the system is simply serialized behind one network call.
- • Thread A holds L1 and blocks acquiring L2; thread B holds L2 and blocks acquiring L1. The dump shows each waiting on a lock the other owns — a two-node cycle, and the resolution is a lock order, not a bigger pool.
- • All workers WAITING on
notEmptywhile the producer already enqueued and signalled before any consumer began waiting: the signal was lost, the queue is non-empty, and every consumer is parked. The dump shows a full queue and idle consumers, which is the signature of a lost wakeup.
- • A dump guarantees the state and stack of each thread at one instant. It does not guarantee those states persisted for even one microsecond after capture.
- • It names the lock owner where the runtime tracks ownership. Ownership is *not* tracked for every synchronization type — many condition and semaphore waits have no owner to report, and the dump will honestly say so.
- • It shows managed-runtime frames. Native frames, kernel wait channels and the actual syscall a thread is parked in may be absent entirely.
- • It says nothing about the past. A thread that spent 99% of the last minute contended can appear perfectly healthy in the instant you sampled.
- • Capture itself may require a global safepoint, so taking a dump adds a pause proportional to thread count — the observation costs the system something.
- • A stalled service is exactly when dumps are most needed and when the safepoint is hardest to reach, because the stalled threads may be in states that delay it.
- • Writing a large dump to disk or a log pipeline during an incident competes with the service for I/O and can worsen the very stall being diagnosed.
- • Misreading idle pool threads WAITING on an empty queue as a symptom, and "fixing" a healthy system.
- • Reading the twenty-two identical blocked stacks and never reading the owner, so the incident is filed as "lock contention" with no cause.
- • Taking exactly one dump and declaring a thread stuck when it was merely sampled mid-operation.
- • Missing a deadlock because ownership is not reported for the primitive in use, so the cycle has to be reconstructed by hand from resource names.
- • Losing the dump: capturing to stdout on a container whose log driver drops under pressure, precisely during the incident.
- • Total stalls with low CPU, where the dump usually names the cause in one reading and no instrumentation was needed in advance.
- • Deadlocks, where the wait-for cycle is literally printed and the fix (an ordering) follows directly — see Lock Ordering.
- • Production systems with no profiler and no lock metrics, because a dump requires nothing to have been set up beforehand.
- • Confirming a pool is the ceiling: all N workers busy and a non-empty queue is unambiguous in a way a utilization gauge is not.
- • Diagnosing CPU-bound slowness, where sampling bias makes dumps systematically point at long I/O rather than at the hot loop actually burning the core.
- • Highly async runtimes, where a handful of OS threads look idle while thousands of tasks are pending — the dump is technically correct and completely uninformative. That is why Task Dumps: When the Threads Look Idle and Nothing Is Moving exists.
- • Very large thread counts, where the safepoint pause and the sheer volume make capture disruptive and reading impractical without aggregation.
- • Blocked count as a fraction of pool size — above roughly half, the service has effectively one thread.
- • Whether the same stack appears in all three dumps taken thirty seconds apart: that is the stuck/working test.
- • Whether a lock owner is reported and what its stack contains — application code means a slow section, a socket frame means I/O under the lock.
- • Presence of a wait cycle: A waits on a lock owned by B, B waits on a lock owned by A.
- • Whether idle workers coexist with a non-empty queue, which is the lost-wakeup signature.
- • The reading skill is not free: state vocabulary, ownership semantics and frame filtering differ per runtime, and every team has to learn theirs.
- • Capture during an incident needs to be a rehearsed procedure with a known destination, or it will not happen when it matters.
- • Aggregating stacks well enough to be readable requires tooling; a raw dump of a thousand threads is a wall of text nobody parses under pressure.
- • Dumps contain call stacks with argument-shaped frames and thread names, which can carry sensitive context into a log pipeline — see What You Just Wrote Into a Log Half the Company Can Read.
- • Continuous lock-wait metrics, which answer the same question without requiring the stall to be in progress but must be instrumented in advance — Hold Time, Wait Time, and the Ratio Between Them.
- • An off-CPU profile, which is a dump taken continuously and aggregated, giving both the waiter and the owner over time at higher cost — Off-CPU Time: The Thing a CPU Profiler Cannot See.
- • A distributed trace with explicit acquire spans, when the waiting matters across service boundaries rather than within a process.
- • For async runtimes, a task dump rather than a thread dump — the thread view is structurally the wrong abstraction there.
Thread dump lab
The frames, thread ids and monitor address are made up to show the shape of the reasoning. Other runtimes print this differently — Go dumps goroutines, Python dumps frames per thread, Node has no equivalent because its work is not on threads at all. What transfers is the method: read states, group the blocked threads by the monitor they name, then find the one thread that does not say “waiting to lock”.
"http-nio-8080-exec-1" #21 daemon prio=5 tid=0x00007f9c1001 nid=0x3b01
java.lang.Thread.State: BLOCKED (on object monitor)
at com.shop.Inventory.reserve(Inventory.java:88)
- waiting to lock <0x00000007a1b2c3d4> (a com.shop.Inventory)
at com.shop.CheckoutService.placeOrder(CheckoutService.java:141)
at com.shop.CheckoutController.post(CheckoutController.java:57)
at org.apache.tomcat.util.threads.TaskThread.run(TaskThread.java:61)What people believe, and what is true
Lots of threads in WAITING means the system is stuck.
A worker pool parked on an empty queue is entirely healthy. Waiting without an owner is usually idleness; blocked with an owner is the alarming one.
Twenty blocked threads means twenty bugs.
It means one bug with twenty witnesses. The single interesting thread is the one that owns the lock.
The dump shows what the program is doing.
It shows what the program was doing at one instant, biased toward slow operations. For "what is it doing", you need repeated samples or a profiler.