Scaling & CachingdebuggingIntermediate

Every night at 00:00 the database falls over for ninety seconds

Symptoms

  • Database CPU hits 100% from 00:00:00 to about 00:01:30 every night; product-page p99 goes from 40 ms to 6 s.
  • Redis hit rate drops from 97% to 11% at exactly midnight, then climbs back over two minutes.
  • Nothing is scheduled at midnight — the nightly jobs run at 03:00.
  • The slow-query log fills with thousands of copies of the same SELECT … FROM products WHERE id = $1, many with the *same* id.
# redis INFO, sampled each second
time        keys      hits/s    misses/s   expired_keys (delta)
23:59:58    184,203   11,920    340        41
23:59:59    184,190   11,880    350        38
00:00:00    1,118     390       12,100     183,072     ← everything expired in one second
00:00:05    9,870     1,240     10,800     0
00:00:20    61,400    6,300     5,900      0
00:01:30    171,050   11,600    410        0

# product cache helper (src/cache/product.ts)
const ttl = secondsUntilMidnight()             // "prices change at midnight, cache must not outlive the day"
await redis.set('product:' + id, JSON.stringify(p), 'EX', ttl)

# pg_stat_activity at 00:00:03 — 412 active backends
SELECT ... FROM products WHERE id = 3382     × 71  (all state=active, all started within 40 ms)
SELECT ... FROM products WHERE id = 9906     × 64
SELECT ... FROM products WHERE id = 1207     × 58

Investigate

Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.

Scheduled jobs
Redis memory and eviction
Traffic at midnight
How the TTL is computed
The cache-miss path
Connection pool