Background Jobs & Queues
Work that does not belong in the request path: deciding what to defer, the queue lifecycle from enqueue to dead-letter, and what happens when producers outrun consumers.
Work that outlives the request that asked for it — and the four guarantees you give up to move it there.
Five questions that decide where work runs — and the reminder that deferring is a cost, not a default.
Enqueue, claim, process, ack, retry, dead-letter — the six-step lifecycle every queue implements, however it spells them.
Ordering, duplicates, retries, visibility timeouts and poison messages — the five properties that differ between every broker you will use.
Delivery will repeat, so the effect must not. How to make a worker safe to run twice, including concurrently.
Cron in a single process is a timer. Cron on three instances is three timers, and the job runs three times.
Somewhere for work that will never succeed, so that one poison message cannot consume the fleet — and a human who is expected to look.
When producers outrun consumers, something has to give. Backpressure is choosing what, instead of letting memory choose for you.
More workers help until the shared dependency saturates, at which point they make everything worse — including the requests still in the path.
The queue is growing. Four possible causes, and the recovery that is right for one of them makes two of the others worse.