Files, File Systems & DescriptorsIntermediate
EMFILE after thirty hours of uptime
Symptoms
Error: EMFILE: too many open filesfromaccept()and, moments later, from opening the log file.- It happens about 30 hours after each deploy; a restart clears it.
- Concurrent client connections are steady at ~40; request rate is unchanged.
- The soft descriptor limit is 1,024.
$ ls /proc/4242/fd | wc -l # Linux
1021
$ cat /proc/4242/limits | grep 'open files'
Max open files 1024 65536 files
$ lsof -p 4242 -a -d 0-99999 | awk 'NR>1 {print $5}' | sort | uniq -c | sort -rn
962 IPv4
30 REG
14 a_inode
6 CHR
5 FIFO
$ ss -tanp | awk '$1=="CLOSE-WAIT"' | grep 'pid=4242' | wc -l
940
$ ss -tanp state close-wait | grep 'pid=4242' | head -3
Recv-Q Send-Q Local Address:Port Peer Address:Port
1 0 10.0.1.12:51734 10.0.3.17:443 users:(("node",pid=4242,fd=812))
1 0 10.0.1.12:51802 10.0.3.17:443 users:(("node",pid=4242,fd=813))
1 0 10.0.1.12:51944 10.0.3.17:443 users:(("node",pid=4242,fd=814))Investigate
Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.
Raise the limit
Legitimate connection count
Descriptor types (`lsof`)
Socket states (`ss`)
The HTTP client code
Log rotation