Practice: Debugging Across Layers
Realistic production symptoms with evidence — top output, socket tables, packet captures, DNS answers. Choose what to investigate; each area reveals a finding, and some are red herrings. Name the layer the fault lives in, find the root cause, then compare your fix.
Debugging challenges
ConcurrencyAdvanced
Every core pinned at 100%, throughput lower than yesterday
All eight cores at 100% since the worker pool was raised from 4 to 32 threads.
SchedulingIntermediate
One core at 100% forever, and the service still answers
Since the last deploy the process sits at exactly 100% CPU (one core) around the clock.
MemoryIntermediate
RSS climbs 200 MB an hour until the OOM killer visits
The ingest service’s RSS grows linearly and it is restarted every ~18 hours with exit code 137.
Virtual MemoryAdvanced
The box is at 3% CPU and everything takes ten seconds
Every process on the host is slow — the API, PostgreSQL, even `ssh` login takes 20 seconds.
ConcurrencyAdvanced
The API stopped answering; the process is alive and idle
All requests time out; the process is alive, at 0.3% CPU, memory normal.
I/OIntermediate
The worker is stuck and `kill -9` does nothing
A batch worker stopped making progress an hour ago.
FilesIntermediate
EMFILE after thirty hours of uptime
`Error: EMFILE: too many open files` from `accept()` and, moments later, from opening the log file.
MemoryIntermediate
Segfault at exactly the same input size, only in worker threads
The parser service crashes with `SIGSEGV` on some documents; the same documents parse fine in the unit tests.
ProcessesIntermediate
Thousands of `<defunct>` processes, then fork fails
`ps` lists thousands of `[worker] <defunct>` entries, all with the same parent.
ThreadsIntermediate
Timers fire late, health checks fail, one core at 100%
The Node.js API intermittently fails its health check although the process is up.
TogetherIntermediate
One slow client makes the whole API slow
Every few minutes the API’s p99 jumps from 40 ms to 8–12 s for all clients, then recovers.
TogetherAdvanced
Connection errors to a healthy upstream at 400 requests per second
A proxy service starts failing calls to its upstream with `connect: cannot assign requested address` (`EADDRNOTAVAIL`) once traffic passes ~400 req/s.
TogetherAdvanced
The producer hangs, the consumer is idle, and nothing is on the wire
An event-ingestion pipeline stops: the producer service’s `send()` calls block and its internal queue fills; the consumer service shows 2% CPU.
TogetherIntermediate
"Too many open files" — and then the health check passes
After 6–8 hours of traffic, the service starts returning 500s; logs show `accept: too many open files` and `EMFILE` from an HTTP client.