Stack & HeapIntermediate

RSS climbs 200 MB an hour until the OOM killer visits

Symptoms

  • The ingest service’s RSS grows linearly and it is restarted every ~18 hours with exit code 137.
  • dmesg shows the OOM killer choosing the process; heap-used in the runtime’s metrics grows at the same rate as RSS.
  • Growth rate tracks the incoming message rate, not wall-clock time.
  • It began after the downstream database migration two weeks ago; the service code has not changed.
$ ps -o pid,rss,vsz,nlwp,cmd -p 4242            # Linux, sampled hourly
  PID    RSS     VSZ NLWP CMD
 4242 1180432 2913220   11 node ingest.js       # 09:00
 4242 1391860 3124512   11 node ingest.js       # 10:00
 4242 1602004 3335808   11 node ingest.js       # 11:00

$ dmesg -T | grep -i oom | tail -1
[Tue 03:12:41] Out of memory: Killed process 4242 (node) total-vm:4210344kB, anon-rss:3902116kB, file-rss:2048kB, shmem-rss:0kB

# heap snapshot diff, 10:00 vs 11:00 (Chrome DevTools, sorted by retained size delta)
Constructor            #New     #Deleted   Size delta   Retained by
Array                  1        0          +198 MB      pendingWrites (closure in BatchWriter)
Object                 1,410,338 1,102,400 +197 MB      (elements of the above)
Buffer                 12       10         +0.3 MB
(closure)              4        4          +0 B

Investigate

Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.

Native memory / allocator fragmentation
Page cache accounting in the cgroup
Heap snapshot diff
Producer and consumer rates
GC configuration
Event listener leak