OS + Networking TogetherAdvanced

The producer hangs, the consumer is idle, and nothing is on the wire

Symptoms

  • An event-ingestion pipeline stops: the producer service’s send() calls block and its internal queue fills; the consumer service shows 2% CPU.
  • Both hosts have idle network links; ping and traceroute between them are normal.
  • It started 20 minutes after the consumer deployed a change that writes each event to a new "audit" table.
# on the CONSUMER host
$ ss -tin state established '( sport = :7000 )'
ESTAB 4194304 0   10.0.2.20:7000  10.0.2.10:53112
	 rcv_space:4194304 rcv_ssthresh:4194304 ... rwnd_limited:100.0%  (recv-q is FULL)

$ cat /proc/$(pgrep consumer)/status | grep State
State:	S (sleeping)
$ cat /proc/$(pgrep consumer)/wchan
futex_wait_queue

# on the PRODUCER host
$ ss -tin state established '( dport = :7000 )'
ESTAB 0 4194304   10.0.2.10:53112  10.0.2.20:7000
	 rtt:0.4/0.1 cwnd:10 unacked:0 notsent:4194304 rwnd_limited:100.0%
# tcpdump on the producer: only window probes every ~60s
14:02:11 IP 10.0.2.10.53112 > 10.0.2.20.7000: Flags [.], seq 1, ack 1, win 0 [TCP ZeroWindowProbe]
14:02:11 IP 10.0.2.20.7000 > 10.0.2.10.53112: Flags [.], ack 1, win 0

Investigate

Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.

Network path between the hosts
Consumer socket receive queue
What the consumer thread is doing
The audit-table change
Producer host resources
Kernel buffer tuning