Async ProcessingoptimizationIntermediate
The notification queue has been "almost caught up" for four days
Symptoms
- Queue depth is 2.4 M and the oldest message is 7.4 hours old; both numbers have risen every day since Monday.
- Order confirmation emails arrive hours after the order; customers call to ask whether the order went through.
- Worker CPU is low (~9% per worker); adding CPU to the worker nodes changed nothing.
- It started on Monday at 09:00, the morning the "order timeline" feature launched.
# queue: notifications (per-minute rates, daily averages) day enqueue/min dequeue/min depth oldest msg age Fri 3,900 4,100 1,200 2 s Mon 8,700 4,150 290,000 41 min ← order-timeline launch 09:00 Tue 8,900 4,180 980,000 3.2 h Wed 9,100 4,160 1,700,000 5.9 h Thu 9,000 4,190 2,400,000 7.4 h # consumers: 8 worker processes × concurrency 1 (prefetch 1) # handler, average 112 ms: # load user prefs (SELECT, indexed) 3 ms # render template 6 ms # sendgrid.send() 98 ms ← network wait; the worker does nothing else meanwhile # ack 5 ms # ceiling: 8 workers × 1 × (60 000 / 112) ≈ 4,285 msg/min # producer change (order-svc, Monday): emits NotificationRequested on every status change # before: OrderPlaced only (1 per order) # after: Placed, Paid, Packed, Shipped (4 per order) — prefs default: email for all four
Investigate
Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.
Worker CPU
The feature launched Monday
Consumer count and concurrency
SendGrid latency
Database
Broker health
Dead-letter queue