OperationsTOOL-SPECIFICGENERALSCALE-SPECIFIC
Operating Queues and Scheduled Work
Depth, oldest message age, consumer throughput, failure rate and dead letters — plus the clock-driven cousin, where duplicate and missed runs live.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The production question
What do you watch on a queue, and what does each signal mean when it moves?
The problem
Asynchronous work is invisible by construction. A queue absorbs a problem quietly for a while and then presents it all at once, usually as a business failure rather than a technical alert.
What teams do first
Put slow work on a queue and it stops affecting users. Alert if the queue gets too deep and otherwise leave it alone.
How it breaks
Depth alone is ambiguous. A deep queue that is draining quickly is fine; a shallow queue whose oldest message is old is broken. The alert fires on the wrong one.
How it breaks in production
- Depth alone is ambiguous. A deep queue that is draining quickly is fine; a shallow queue whose oldest message is old is broken. The alert fires on the wrong one.
- Consumers can be running, healthy, and processing nothing — subscribed to the wrong topic, stuck on a poison message, or blocked on a dependency (The Backlog Arithmetic: Four Levers and a Drain Time in Observability).
- Failures move sideways into a dead letter queue and disappear from the metric everyone watches (Dead Letter Queues Are an Operation).
- The user-visible symptom is delay, not error: the email never arrives, the order sits unprocessed. No dashboard is red.
- The same class of failure applies to scheduled jobs, where the queue is a clock: runs that overlap, runs that never happen, and runs that happen twice.
Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A queue is a buffer between a producer rate and a consumer rate. It is healthy when consumption is at least production over any window that matters, and it hides the difference until it cannot.
- The two fundamental signals are depth — how much work is waiting — and oldest message age — how long the oldest waiting item has been waiting. Age is the one that maps to user experience; depth is the one that maps to memory and storage.
- Throughput and failure rate explain the age: consumers processing nothing, or failing everything and retrying, both produce a rising age with very different fixes.
- Retries interact with visibility timeouts and acknowledgement: a consumer that is slower than the visibility timeout has its message redelivered while it is still working, producing duplicate processing that looks like a bug in the application.
- Which means: at-least-once delivery is the normal case, and idempotent consumers are an operational requirement, not a nicety (Idempotency in Architecture).
- A scheduled job is a producer with a clock instead of an event. Its failure modes are the clock's: two runs at once because the previous one has not finished, no run at all because the scheduler was down at the moment, and two runs because of a time change (Timezone and DST Failures).
Five signals, and what each one means when it moves
A queue with only a depth alert is a queue with one ambiguous signal. Together these five distinguish causes that need completely different responses.
- Age flat and high with healthy throughput means the front of the queue is stuck while the rest flows — typical of ordered partitions with a poison message.
- Zero dead letters for a long period usually means dead-lettering is not configured, not that nothing has failed.
| Signal | What it means | Rising means | Alert on it? |
|---|---|---|---|
| Depth | Work waiting | Producers outpacing consumers, or consumers stopped | Supporting signal — ambiguous alone |
| Oldest message age | How long the front of the line has waited | Something is not being processed at all | Yes — this is the user-facing one |
| Dequeue rate | Actual consumer throughput | Falling to zero means consumers are alive but idle | Yes, as absence: zero with non-zero depth |
| Failure rate | Messages failing processing | A poison message, a bad deploy, or a sick dependency | Yes, as a ratio of throughput |
| Dead letter count | Work that has given up | Anything above zero is work that failed permanently | Yes — always (Dead Letter Queues Are an Operation) |
What the operator sees, and where the work goes
TOOL-SPECIFICWhere retry lives differs: some brokers redeliver on unacknowledged messages after a visibility timeout, some require the consumer to re-enqueue explicitly, and log-based systems have no per-message retry at all — the consumer manages its own offset and error handling. The operational questions are identical; the mechanism is not.
Drawing the path makes the sideways exit obvious. Most queue dashboards show the horizontal flow and omit the branch, which is where failed work quietly accumulates.
UserLLMAgentToolDataDecisionHumanGuardrail
Scheduled jobs: a queue with a clock
Cron-style scheduling looks simpler than a queue and has more failure modes, because the trigger is time and time is not as reliable an input as people assume.
The three that cause most incidents are duplicate execution, missed execution and overlap. All three are invisible unless you assert on runs rather than on failures.
Scheduled job failure modes
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Previous run has not finished when the next is due | Two runs process the same records concurrently | No single-flight lock; the schedule assumes the job is faster than its interval | Take a lock with a lease; skip or queue the overlapping run, and alert on skips |
| Scheduler restarted or node replaced at the trigger moment | A run simply never happened | Failure alerts only fire for runs that started | Assert expected-versus-actual runs; alert on absence (Job Scheduler Reliability) |
| Job retried after partial completion | Duplicate side effects — double emails, double charges | The run was not idempotent or resumable | Checkpoint progress; key side effects on a stable idempotency key (Idempotency Keys: The Mechanism in Backend Engineering) |
| Schedule expressed in a local timezone | The run happens twice, or not at all, on one day of the year | A daylight saving transition repeats or skips a local hour (Timezone and DST Failures) | Schedule in UTC and translate for display only |
| Job runtime grows with data volume | Runs start overlapping months after deployment, with no change | The interval was chosen when the dataset was small | Alert on run duration as a fraction of the interval, before it reaches one |
How to do it properly
Most important first.
- Alert on oldest message age against a threshold derived from what the work promises, and use depth as a supporting signal rather than the primary one.
- Make every consumer idempotent, keyed on something stable from the message, and assume redelivery (Job Idempotency in Backend Engineering).
- Retry with bounded attempts and backoff with jitter, then dead-letter — never retry forever in place.
- Bound the queue or shed load deliberately at the producer; an unbounded queue converts a throughput problem into a memory or storage problem (Bounded vs Unbounded Queues in Concurrency).
- For scheduled work: enforce single-flight with a lock so runs cannot overlap, alert on a missed run rather than only on a failed one, keep each run resumable, and schedule in UTC (Production Time Is UTC).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Blast radius if this is wrongOne tenant
One testEveryone
What contains it
Per-tenant or per-key partitioning contains a poison message to one stream; bounded retries and dead-lettering contain a failing consumer. An unbounded, unordered queue with infinite retries contains nothing and spreads to the downstream dependency.
What can go wrong
Failure modes, including of the mitigation
- Consumers alive and consuming nothing; the health check checks the process, not the work.
- A poison message blocks an ordered partition, so one bad item stops a whole stream.
- A scheduled job overlaps itself, and two runs process the same records concurrently.
- A scheduled run is skipped during a deploy or a scheduler restart, and nothing notices because the alert fires on failures only.
- The backlog is drained by deleting messages, discarding work that mattered because nobody could tell what it contained.
Misreads this invites
- "The queue is empty, so everything is fine." An empty queue with no consumers looks identical to an empty queue with fast consumers. Watch throughput too.
- "Depth is the metric." Age is the metric that maps to a user waiting; depth maps to resource usage.
- "Exactly-once delivery." Practically every broker gives at-least-once, and exactly-once *processing* is achieved by idempotent consumers, not by the broker.
- "The job is scheduled, so it runs." It runs when the scheduler is healthy, the previous run has finished, and the clock did what you expected (Job Scheduler Reliability).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
How you know it worked
- A dashboard per queue: depth, oldest message age, enqueue rate, dequeue rate, failure rate, dead letter count.
- An alert on age that has fired at least once for a real cause, so you know it works and where it routes.
- For scheduled jobs: a record of expected versus actual runs, so a missed run is visible as an absence.
How you get back
- Queues make rollback interesting: messages produced by a new version may be consumed by an old one after a rollback, so message schema changes need the same expand/contract discipline as database schema changes (Version Coexistence: N and N+1, in Both Directions).
- Pausing consumers is the safest immediate lever during a bad drain — the work stays queued while you decide, provided retention outlasts the incident.
- Purging a queue is irreversible. Snapshot the messages somewhere first, even under pressure, because "what was in there" becomes the postmortem's central question.
What to automate, and what stays human
- Automate: age and failure alerting, consumer scaling from queue signals, backoff and dead-lettering, and a heartbeat check on long-running jobs.
- Automate the missed-run check for scheduled work — an assertion that a run happened within the expected window, which catches the failure a job cannot report itself.
- Keep human: draining or purging a backlog, replaying dead letters in bulk, and deciding whether delayed work is still worth doing when it finally runs.
What this costs
- Deeper queues absorb larger bursts and hide problems longer; shorter ones surface problems earlier and shed work sooner (Load Shedding).
- Strict ordering gives simpler application reasoning and lets one bad message block a partition. Ordering per key is usually the workable middle.
- Idempotent consumers require a dedupe store, which is real state and real cost, in exchange for surviving the redelivery that will definitely happen.
- Aggressive consumer autoscaling drains backlogs fast and can crush the downstream dependency the backlog was caused by.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICBrokers differ in what they even expose: a log-based system such as Kafka reports consumer lag in offsets per partition, while a hosted queue reports approximate depth and age of the oldest message. Ordering, redelivery and dead-letter semantics differ with them — check what your broker guarantees rather than assuming the general model.
- GENERALDepth, age, throughput, failure rate and dead letters are the five questions for any asynchronous work system, including a database-backed job table.
- SCALE-SPECIFICAt low volume a job table with a lock is simpler and more debuggable than a broker, and most of this lesson still applies to it. Brokers earn their operational cost at throughput, partitioning and multi-consumer fan-out.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
Backendjob-queuesqueue-backlogqueue-semanticsbackground-jobsscheduled-jobsjob-idempotencybackpressure
Domains that do not exist yet
- — Distributed Systems — why exactly-once delivery is not on offer, and what at-least-once forces on every consumer you write.