Scaling & CachingdebuggingIntermediate
Every night at 00:00 the database falls over for ninety seconds
Symptoms
- Database CPU hits 100% from 00:00:00 to about 00:01:30 every night; product-page p99 goes from 40 ms to 6 s.
- Redis hit rate drops from 97% to 11% at exactly midnight, then climbs back over two minutes.
- Nothing is scheduled at midnight — the nightly jobs run at 03:00.
- The slow-query log fills with thousands of copies of the same
SELECT … FROM products WHERE id = $1, many with the *same* id.
# redis INFO, sampled each second
time keys hits/s misses/s expired_keys (delta)
23:59:58 184,203 11,920 340 41
23:59:59 184,190 11,880 350 38
00:00:00 1,118 390 12,100 183,072 ← everything expired in one second
00:00:05 9,870 1,240 10,800 0
00:00:20 61,400 6,300 5,900 0
00:01:30 171,050 11,600 410 0
# product cache helper (src/cache/product.ts)
const ttl = secondsUntilMidnight() // "prices change at midnight, cache must not outlive the day"
await redis.set('product:' + id, JSON.stringify(p), 'EX', ttl)
# pg_stat_activity at 00:00:03 — 412 active backends
SELECT ... FROM products WHERE id = 3382 × 71 (all state=active, all started within 40 ms)
SELECT ... FROM products WHERE id = 9906 × 64
SELECT ... FROM products WHERE id = 1207 × 58Investigate
Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.
Scheduled jobs
Redis memory and eviction
Traffic at midnight
How the TTL is computed
The cache-miss path
Connection pool