The LLM is unavailable.Investigate with fundamentals.
Operating SystemsAdvanced

Memory usage keeps increasing

Page: catalog-api pods are OOM-killed every 5–7 hours. Working-set memory climbs in a straight line from 380 MiB after start to the 2 GiB limit, then the pod restarts and the line starts again. Traffic has been flat for a month.

What you know

  • Node.js 20, 6 replicas, container limit memory: 2Gi, started with --max-old-space-size=1536. Restarting a pod resets it completely.
  • Three weeks ago a "response memoization" middleware was added to cut CPU on GET /products/search, keyed by the request URL.
  • The mobile client’s HTTP layer appends a cache-busting _=<timestamp ms> parameter to every request it makes.
  • Available: container memory and OOM events, --trace-gc output in the pod logs, a heap snapshot taken at t+4h from one pod, socket and file-descriptor counts.
# kubectl describe pod catalog-api-6d9f-x7k2
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
    Restart Count:  4
    Limits:   memory: 2Gi

# node dmesg
Memory cgroup out of memory: Killed process 31877 (node) total-vm:2412304kB, anon-rss:2084112kB

# container_memory_working_set_bytes, one pod (MiB)
t+0h  380    t+1h  640    t+2h  910    t+3h 1 190    t+4h 1 470    t+5h 1 750    t+5h40  OOM

# --trace-gc, t+4h  (heap before -> after, in MB)
[Mark-Compact (reduce) 1462.1 (1498.3) -> 1458.9 (1497.6) MB, 812 ms]
[Mark-Compact (reduce) 1466.4 (1502.0) -> 1463.0 (1501.5) MB, 831 ms]

# heap snapshot, t+4h, sorted by retained size
Constructor        Count        Shallow      Retained
Map                    1        48 B         1 291 MB    ← middleware/memo.js  const cache
(string)       1 482 913        …              612 MB
Object         1 482 902        …              540 MB

# sample of keys in cache
"GET /products/search?q=lamp&_=1724570311204"
"GET /products/search?q=lamp&_=1724570311391"
"GET /products/search?q=lamp&_=1724570312007"

# inside the pod, t+4h
$ ss -s | head -2
Total: 118
TCP:   61 (estab 42, closed 12, orphaned 0, timewait 7)
$ ls /proc/1/fd | wc -l
88

Investigate

For each area: first say why you would check it (reveal the reasoning), then look. Commit to a root cause when you are confident.

Container memory limit and OOM events
GC logs
Heap snapshot and allocation profile
Connection and file-descriptor leak
The caching middleware code
Log buffering
Heap vs RSS