intermediate

The queue that adding workers did not fix

Traffic is flat, throughput halves, and tripling the worker count changes almost nothing.

The page

WARN · notification-worker · oldest message age 2,410 s (threshold 300 s) · queue depth 184,000 and rising

Timeline — in the order it was observed

Observation order is not causal order. The first thing anyone noticed is rarely the first thing that happened.

  1. 10:15Support reports order-confirmation emails arriving 20 to 30 minutes late. Nothing is failing — the emails do arrive.support queue
  2. 10:22Queue depth alert fires. Depth is 184,000 and climbing steadily.queue_depth
  3. 10:24Producer rate is checked: 340 messages per second, flat all week. No traffic anomaly.messages_published
  4. 10:26Completion rate is 152 messages per second, down from a steady 355. Workers are not failing — they are finishing fewer jobs.messages_acked
  5. 10:31Worker count is scaled from 12 to 36. Completion rate rises to 198 per second, then flattens. The backlog keeps growing.incident channel
  6. 10:44Worker CPU across all 36 instances is 11%. Every worker slot is occupied and almost no CPU is being used.cpu_utilization
  7. 10:58A job trace shows 8.4 s of a 9.1 s job inside a single call to the customer profile API.job trace
  8. 11:20Profile API owners confirm a schema migration started at 09:50 that added a sequential scan to the lookup path.upstream team
The system
~340 msg/sper messageorder-api (producer)notifications queuenotification-worker ×12Customer profile APIEmail provider
Pull up evidence · 0/6 opened

Most of this evidence is consistent with several explanations. Keep going until something narrows it.

The Backlog Arithmetic: Four Levers and a Drain TimeDepth Is Not an Emergency; Age IsSix Queue Signals, Two That Wake You UpTwenty Workers, All Busy, Five Hundred WaitingLittle's Law as Working IntuitionComputing or Waiting?Timeouts: The Latency Contract Nobody Writes DownSaturation: The Reading Utilization Cannot Give YouThroughput: The Number That Means Nothing Without a Latency Bound