Message Queues

Synchronous A → B is the simplest integration: one call, one answer, one place to look when it fails. It couples A’s availability to B’s and A’s latency to B’s. A → Queue → B decouples both: A’s request completes when the broker has persisted the message, and B consumes at its own rate. What A loses is the answer — it does not learn whether B succeeded — and what the system gains is a new place for work to silently pile up.

The queue is a buffer, and a buffer is only useful if the consumer eventually catches up. A queue in front of a consumer that is permanently slower than the producer is not decoupling; it is a delayed outage with a growing memory bill. Backpressure covers what to do when that happens.

A consumer receives a message and, when finished, acks it; the broker then deletes it. If the consumer crashes first, the message must come back. In SQS this is the visibility timeout: the message is hidden for, say, 30 s after delivery and reappears if no ack arrives. In RabbitMQ an unacked message is redelivered when the channel closes. Either way, delivery is at least once, and the consumer must be idempotent — the same rule as in Event-Driven Architecture.

A visibility timeout shorter than the work is a classic bug: a 45 s job on a 30 s timeout is delivered to a second worker at second 31, both finish, and the effect happens twice. Set the timeout above the p99 of the handler, or extend it from inside the handler while working. Retries need backoff with jitter, otherwise a broker outage ends with every consumer hammering the same dependency at the same instant.

A poison message fails every time — malformed payload, a bug in the handler for one input, a record that no longer exists. Without a limit it is retried forever and blocks the workers that keep picking it up. After N attempts (3–5 is common) the broker moves it to a dead-letter queue (DLQ). A DLQ is not a bin: alert on its depth, inspect the messages, fix the cause, and redrive them. A DLQ that nobody reads is data loss with extra steps.

A queue delivers each message to one consumer: ten workers on one queue share the work (competing consumers). A topic delivers each message to every subscription: pub/sub. A consumer group combines both — the group as a whole receives every message once, and members of the group split it. Kafka builds this in; with RabbitMQ you bind one queue per consumer to a fanout exchange; with AWS you subscribe several SQS queues to one SNS topic. The abstractions in a Queue and a Deque are the mental model: FIFO hand-off with a head that consumers pull from.

Conceptually there are three broker shapes. A RabbitMQ-style queue deletes on ack, routes richly (exchanges, routing keys, priorities), and is good for task distribution with per-message acks. SQS is the same shape as a managed, effectively unbounded service with visibility timeouts and a built-in DLQ; standard queues are unordered and may duplicate, FIFO queues order within a message group at lower throughput. A Kafka-style log never deletes on ack — consumers keep an offset, many groups read the same data, and history can be replayed; it is the right choice when several teams consume the same stream or when you need to rewind. Kafka-Style Logs: Topics, Partitions, Offsets covers it. Redis streams (Redis: Data Structures, Not a Cache) give a small log with consumer groups when you already run Redis and the volume is modest.