OS + Networking TogetherIntermediate

"Too many open files" — and then the health check passes

Symptoms

  • After 6–8 hours of traffic, the service starts returning 500s; logs show accept: too many open files and EMFILE from an HTTP client.
  • Restarting the process fixes it for another 6–8 hours.
  • The load balancer’s health check keeps passing right up to the crash, so traffic is never drained.
$ ls /proc/$(pgrep -f api-server)/fd | wc -l
65536
$ ulimit -n
65536

$ ss -tanp | awk '{print $1}' | sort | uniq -c
   61 ESTAB
64890 CLOSE-WAIT
   11 LISTEN

$ ss -tan state close-wait | head -3
CLOSE-WAIT 1 0  10.0.3.4:41822  10.0.5.9:443
CLOSE-WAIT 1 0  10.0.3.4:41830  10.0.5.9:443

# app log, hours earlier
WARN upstream 10.0.5.9:443 returned 503, retrying
WARN upstream 10.0.5.9:443 returned 503, retrying

Investigate

Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.

The `ulimit -n` value
What the open descriptors are
The code path that talks to 10.0.5.9
Upstream 503 rate
Memory growth
Why the health check kept passing