IP, IPv4, IPv6, Subnets, NAT & PortsAdvanced

Outbound calls fail from the whole cluster whenever traffic peaks

Symptoms

  • During peak hours, services across all nodes see sporadic connect: connection timed out to external APIs.
  • The same APIs answer instantly from a laptop outside the cluster.
  • Failures correlate with request volume, not with any particular destination.
  • Some connections are reset mid-request rather than failing to connect.
# on a worker node (Linux)
$ dmesg | tail -3
[812031.117] nf_conntrack: nf_conntrack: table full, dropping packet
[812031.121] nf_conntrack: nf_conntrack: table full, dropping packet

$ sysctl net.netfilter.nf_conntrack_max net.netfilter.nf_conntrack_count
net.netfilter.nf_conntrack_max = 65536
net.netfilter.nf_conntrack_count = 65536

$ conntrack -L 2>/dev/null | awk '{print $4}' | sort | uniq -c | sort -rn | head -3
  52210 TIME_WAIT
   9105 ESTABLISHED
   3811 SYN_SENT

$ sysctl net.netfilter.nf_conntrack_tcp_timeout_time_wait
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120

# the cloud NAT gateway metrics for the same window
ErrorPortAllocation   spikes to 1,400/min at 17:00–17:20
ConnectionAttemptCount ~ 9,800/s across the cluster

Investigate

Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.

The kernel log
What fills the table
Why so many new connections
The cloud NAT gateway
The external APIs
DNS
The mid-request resets