Virtual Memory & PagingAdvanced
The box is at 3% CPU and everything takes ten seconds
Symptoms
- Every process on the host is slow — the API, PostgreSQL, even
sshlogin takes 20 seconds. - CPU is almost idle but load average is 30 on 8 cores.
topshows highwaand dozens of processes in stateD.- The disk activity light is solid; the SSD is doing constant I/O although nobody deployed anything.
- It started after the JVM heap was raised to 12 GB "because we had spare RAM".
$ vmstat 1 5 # Linux
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 24 7912340 118204 1120 210400 48120 51220 49300 51400 9100 12000 2 4 8 86 0
1 27 7940112 116980 1104 208112 50200 52880 51100 53000 9300 11800 1 3 9 87 0
$ free -m
total used free shared buff/cache available
Mem: 16000 15650 118 4 232 210
Swap: 8192 7750 442
$ top -b -n1 -o RES | head -9
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5120 app 20 0 14.1g 11.2g 8m D 1.2 70.0 91:12 java -Xmx12g
2210 pg 20 0 2.9g 2.1g 1.8g D 0.4 13.1 40:08 postgres: checkpointer
4242 app 20 0 1.4g 0.6g 4m D 0.3 3.9 12:41 node api.js
871 root 20 0 0.2g 0.0g 1m D 0.0 0.1 0:03 sshdInvestigate
Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.
CPU profile of the slow processes
Disk health
`vmstat` swap columns
Who owns the resident memory
The load average
`vm.swappiness` and overcommit settings