← PracticeIntermediateCoordination

The Database Falls Over On the Hour

Pull up the evidence one item at a time, commit to a diagnosis, and only then see the schedule that actually ran.

What was reported

Every hour, at exactly :00, our database CPU goes to 100% for about forty seconds and API p99 goes from 60ms to 9s. Then it recovers completely on its own. Nothing is scheduled at :00 — we checked every cron in the estate. Traffic at :00 is identical to traffic at :59. The cache hit rate chart has a vertical cliff at the top of every hour.
1async function getPricingRules(): Promise<Rules> {
2  const hit = await cache.get('pricing:rules')
3  if (hit) return JSON.parse(hit)
4  const rules = await db.query(expensivePricingQuery)   // 2.1s, seq scan over 40M rows
5  await cache.set('pricing:rules', JSON.stringify(rules), { ttl: 3600 })
6  return rules
7}

Evidence

Nothing here is labelled as relevant. Some of it is not.

What is actually happening?