intermediate
The service that gets slower every four hours
Latency spikes worsen through the day, a restart fixes it, and by evening it is back.
The page
WARN · recommendations-api · pod restart count 4 in 12 h (OOMKilled) · p99 1,840 ms (threshold 900 ms)
Timeline — in the order it was observed
Observation order is not causal order. The first thing anyone noticed is rarely the first thing that happened.
- Tue 09:14v2.4 deployed. Adds per-user recommendation personalisation. Passes canary and load tests.deploy marker
- Tue 15:40First latency spike alert. p99 touches 1.9 s for four minutes, then recovers on its own. Nobody investigates a self-healing blip.request_duration
- Tue 21:02Two more spikes, closer together. On-call notes traffic was up 18% that day and attributes it to load.incident channel
- Wed 03:30A pod is OOMKilled and restarts. Latency returns to baseline immediately afterwards.kubernetes events
- Wed 11:20Restart count alert fires: four OOMKills in twelve hours across the fleet.restart_count
- Wed 11:35Resident memory plotted over 24 h shows a sawtooth: a steady climb, a vertical drop at each restart, then the same climb again.container_memory_rss
- Wed 11:52Garbage-collection time share has risen from 1.2% to 14% of wall clock, and the latency spikes line up with collection cycles.gc_time_fraction
- Wed 12:30Two heap snapshots four hours apart are compared. One map has grown by 210 MB and holds every user id seen since process start.heap diff
The system
Pull up evidence · 0/7 opened
Most of this evidence is consistent with several explanations. Keep going until something narrows it.
Memory Leaks: Growth That Does Not Come BackLeak or Unbounded Cache? The Question That Picks the FixGarbage Collection: Pause, Throughput, Footprint — Pick TwoAllocation Rate Is a Cost Even Without a LeakReading Memory: RSS, Heap, Working Set and the Number on Your DashboardTail Latency: Why p50 Being Fine Does Not Help"What Changed?" — Deploy Markers and the Invisible DeploysCorrelation Is Not the Root CauseLoad Testing: What Question Is This Test Answering?