Concurrency, Synchronization & DeadlocksAdvanced

The API stopped answering; the process is alive and idle

Symptoms

  • All requests time out; the process is alive, at 0.3% CPU, memory normal.
  • A TCP connect to the port succeeds but no HTTP response ever arrives.
  • It happened right after a burst of concurrent cache refreshes; a restart fixes it for a while.
  • The database shows no active queries from this service.
$ ss -ltn 'sport = :8080'                             # Linux
State   Recv-Q  Send-Q  Local Address:Port
LISTEN  129     128     0.0.0.0:8080          # accept backlog full: nobody is accepting

$ ps -L -o tid,stat,wchan:20,comm -p 4242 | head -6
  TID STAT WCHAN                COMMAND
 4242 Sl   futex_wait_queue     api
 4250 Sl   futex_wait_queue     worker-1
 4256 Sl   futex_wait_queue     worker-7
 4258 Sl   futex_wait_queue     worker-9

$ jstack 4242 | tail -12
"worker-1": waiting to lock <0x00000007a1c0e2f0> (a CacheLock)
    which is held by "worker-7"
    at Cache.refresh(Cache.java:88)
    - locked <0x00000007a1c0e410> (a StatsLock)
"worker-7": waiting to lock <0x00000007a1c0e410> (a StatsLock)
    which is held by "worker-1"
    at Stats.record(Stats.java:41)
    - locked <0x00000007a1c0e2f0> (a CacheLock)
Found 1 deadlock.
"worker-2" … "worker-32": waiting to lock <0x00000007a1c0e2f0> (a CacheLock)

Investigate

Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.

Network and load balancer
The database
Blocked I/O in the kernel (`D` state)?
The thread dump
GC pause
Lock acquisition order in the code