Debugging Concurrency
Why concurrency bugs resist ordinary debugging, and what works anyway: lock wait metrics, thread and async task dumps, contention profiling, race detectors, deterministic replay, and stress testing designed to shake out schedules that normal runs never produce.
Request rate, error rate and duration describe the work. They say nothing about whether forty threads are asleep in front of one lock. Concurrency observability adds a second axis — where the work is *waiting* — and there are exactly six signals worth the cardinality.
Q · Which signals tell me my system is concurrency-limited rather than slow?
A lock held for 2ms with a p99 wait of 500ms is contended. Neither number says so on its own — 2ms is a fine critical section and 500ms could be a slow dependency. The ratio is the signal, and it is the one number that tells you whether to shrink the section or reduce the arrivals.
Q · Given a lock's hold time and wait time, how do I tell contention from slow work?
A snapshot of every thread's state and stack. Twenty threads parked in the same lock frame is not twenty problems — it is one, and the dump names it. The skill is reading state and stack together, and knowing that one dump is a photograph while two are a story.
Q · The service is stalled and CPU is near zero. What does a snapshot of every thread tell me?
Four OS threads, all idle. Eleven thousand tasks, all pending. A thread dump reports a healthy process; the truth is that every task is parked on a downstream that stopped answering. Async runtimes need a dump of the tasks, because the thread is no longer the unit of work.
Q · My async service is stalled but every OS thread is idle. Where do I look?
A CPU profiler samples threads that are on a processor. A thread waiting for a lock is not on a processor, so it contributes nothing to the profile — and the flame graph of a fully contended service looks empty. Concurrency profiling measures the time you were *not* running, and attributes it to a stack.
Q · The flame graph is nearly empty and the service is slow. What is a CPU profile structurally unable to show me?
A race detector watches memory accesses and reports two that conflict with no happens-before edge between them. That is a *data race* — a precisely defined thing. It is not the same as a race condition, and the gap between the two is where the bugs that survive a clean detector run live.
Q · A detector reports my program is race-free. What class of bug does that actually rule out?
You add a log line to find out what is happening, and the bug stops happening. That is not bad luck — the log line took a lock, allocated, and did a syscall, which reordered the schedule and closed the window. Concurrency bugs resist the normal debugging loop because the loop's first step perturbs the thing being measured.
Q · Why does adding a print statement make my concurrency bug disappear, and what do I do instead?
If you record every nondeterministic input — the order threads were scheduled, which lock was granted to whom, what the clock said, what came back from the network — you can replay the failing execution exactly. It turns a one-in-forty-thousand bug into a file. It is also not free, and knowing what it costs is half the lesson.
Q · Can I capture a failing interleaving well enough to run it again on demand?
A concurrency test that passes is a statement about one schedule out of an astronomical number. To learn anything, run the same scenario thousands of times with randomized scheduling, injected delays and load — deliberately manufacturing the schedules a normal run will not produce until it is in production.
Q · My concurrency test passes. What have I actually learned?