intermediate

The service that gets slower every four hours

Latency spikes worsen through the day, a restart fixes it, and by evening it is back.

The page

WARN · recommendations-api · pod restart count 4 in 12 h (OOMKilled) · p99 1,840 ms (threshold 900 ms)

Timeline — in the order it was observed

Observation order is not causal order. The first thing anyone noticed is rarely the first thing that happened.

  1. Tue 09:14v2.4 deployed. Adds per-user recommendation personalisation. Passes canary and load tests.deploy marker
  2. Tue 15:40First latency spike alert. p99 touches 1.9 s for four minutes, then recovers on its own. Nobody investigates a self-healing blip.request_duration
  3. Tue 21:02Two more spikes, closer together. On-call notes traffic was up 18% that day and attributes it to load.incident channel
  4. Wed 03:30A pod is OOMKilled and restarts. Latency returns to baseline immediately afterwards.kubernetes events
  5. Wed 11:20Restart count alert fires: four OOMKills in twelve hours across the fleet.restart_count
  6. Wed 11:35Resident memory plotted over 24 h shows a sawtooth: a steady climb, a vertical drop at each restart, then the same climb again.container_memory_rss
  7. Wed 11:52Garbage-collection time share has risen from 1.2% to 14% of wall clock, and the latency spikes line up with collection cycles.gc_time_fraction
  8. Wed 12:30Two heap snapshots four hours apart are compared. One map has grown by 210 MB and holds every user id seen since process start.heap diff
The system
StorefrontLoad balancerrecommendations-api ×6product-dbRedis
Pull up evidence · 0/7 opened

Most of this evidence is consistent with several explanations. Keep going until something narrows it.