OS + Networking TogetherAdvanced
Connection errors to a healthy upstream at 400 requests per second
Symptoms
- A proxy service starts failing calls to its upstream with
connect: cannot assign requested address(EADDRNOTAVAIL) once traffic passes ~400 req/s. - The upstream reports no errors and low latency; a
curlfrom the proxy host to the upstream works. - The failures come in waves lasting about a minute, then clear briefly, then return.
$ ss -s TCP: 29418 (estab 212, closed 28906, orphaned 3, timewait 28871) $ ss -tan state time-wait '( dport = :9200 )' | wc -l 28190 $ sysctl net.ipv4.ip_local_port_range net.ipv4.ip_local_port_range = 32768 60999 $ sysctl net.ipv4.tcp_tw_reuse net.ipv4.tcp_tw_reuse = 0 # proxy log dial tcp 10.0.9.5:9200: connect: cannot assign requested address dial tcp 10.0.9.5:9200: connect: cannot assign requested address
Investigate
Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.
Upstream health and capacity
Firewall and security groups
`ss -s` socket summary
How the proxy opens upstream connections
File descriptor limit
DNS resolution of the upstream