OS + Networking TogetherIntermediate

One slow client makes the whole API slow

Symptoms

  • Every few minutes the API’s p99 jumps from 40 ms to 8–12 s for all clients, then recovers.
  • CPU on the API hosts is under 20% during the incident; no errors are logged.
  • The spikes correlate with a mobile client on a poor connection downloading a large report export.
$ ss -tino state established '( sport = :8080 )' | head
ESTAB 0      3145728  10.0.4.12:8080  198.51.100.77:51203
	 cubic wscale:7,7 rto:2280 rtt:1140/210 ato:40 mss:1448 cwnd:4 ssthresh:7
	 bytes_sent:52428800 bytes_retrans:1245280 unacked:4 rcv_space:14600 notsent:3087500
ESTAB 0      0        10.0.4.12:8080  10.0.7.31:44821   ... rtt:0.3/0.1 cwnd:10
ESTAB 0      0        10.0.4.12:8080  10.0.7.32:44830   ... rtt:0.4/0.1 cwnd:10

$ curl -s localhost:9090/metrics | grep -E 'threads_(busy|max)'
http_threads_busy 32
http_threads_max  32

# app log (one line per second)
[pool] all 32 worker threads busy; queue depth 417

Investigate

Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.

Server CPU and load average
Database latency
The slow client’s connection in `ss -tin`
How the handler writes the response
Thread pool size and queue
Load balancer health checks
Kernel socket buffer sizes