OS + Networking TogetherAdvanced
The producer hangs, the consumer is idle, and nothing is on the wire
Symptoms
- An event-ingestion pipeline stops: the producer service’s
send()calls block and its internal queue fills; the consumer service shows 2% CPU. - Both hosts have idle network links;
pingandtraceroutebetween them are normal. - It started 20 minutes after the consumer deployed a change that writes each event to a new "audit" table.
# on the CONSUMER host $ ss -tin state established '( sport = :7000 )' ESTAB 4194304 0 10.0.2.20:7000 10.0.2.10:53112 rcv_space:4194304 rcv_ssthresh:4194304 ... rwnd_limited:100.0% (recv-q is FULL) $ cat /proc/$(pgrep consumer)/status | grep State State: S (sleeping) $ cat /proc/$(pgrep consumer)/wchan futex_wait_queue # on the PRODUCER host $ ss -tin state established '( dport = :7000 )' ESTAB 0 4194304 10.0.2.10:53112 10.0.2.20:7000 rtt:0.4/0.1 cwnd:10 unacked:0 notsent:4194304 rwnd_limited:100.0% # tcpdump on the producer: only window probes every ~60s 14:02:11 IP 10.0.2.10.53112 > 10.0.2.20.7000: Flags [.], seq 1, ack 1, win 0 [TCP ZeroWindowProbe] 14:02:11 IP 10.0.2.20.7000 > 10.0.2.10.53112: Flags [.], ack 1, win 0
Investigate
Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.
Network path between the hosts
Consumer socket receive queue
What the consumer thread is doing
The audit-table change
Producer host resources
Kernel buffer tuning