EventsGENERALSIMPLIFIEDDATABASE-SPECIFIC

Writing Event Consumers

A consumer is a program that will see every message twice, some out of order, and one that poisons it.

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.

The question

What does a correct event consumer have to handle that a request handler does not?

The requirement

We publish OrderPlaced. A new service must react to it by creating a fulfilment record, and it must not create two.

The obvious build

Subscribe, do the work, ack. It is a function that runs when a message arrives — simpler than an HTTP handler, because there is no response to build.

Why it breaks

The consumer crashes after creating the fulfilment record and before acking. The broker redelivers, and the order is fulfilled twice (At-Least-Once Delivery).

How it breaks in production

The consumer crashes after creating the fulfilment record and before acking. The broker redelivers, and the order is fulfilled twice (At-Least-Once Delivery).
One malformed message fails forever. The consumer retries it, blocks its partition, and every message behind it stops — a single bad payload halts the whole capability (Dead-Letter Queues).
OrderCancelled is consumed before OrderPlaced because they landed on different partitions, so the cancellation applies to nothing and the order ships.
The consumer scales to ten instances and two of them process two updates for the same order concurrently, so the older one wins by finishing last.
A downstream dependency is down; the consumer retries in a tight loop and turns a degraded dependency into a dead one (Retry Storms).
The consumer falls two hours behind and nothing alerts, because the only metric anyone added was error rate, and there are no errors.

Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale

What is actually happening

Delivery is at-least-once in essentially every practical system, because the acknowledgement travels separately from the effect. The consumer's job is to make repeated delivery harmless, not to prevent it (Job Idempotency).
Ordering is guaranteed only within whatever the broker calls a partition or ordering key, and only if you actually publish related events with the same key. Across keys there is no order at all.
The ack point is the whole correctness question. Ack before the work and a crash loses the event; ack after the work and a crash duplicates it. There is no third option, so choose duplication and defend against it.
A consumer group is a unit of independent progress. Two groups on one topic each get every message; two instances in one group split the messages. Getting these confused is how a reaction runs twice or half the time.
Retry has to be bounded and off the hot path. Unbounded in-place retry blocks everything behind it; the standard shape is a few immediate attempts, then a delay queue, then a dead-letter queue with the payload and the error preserved.
A consumer's lag — the age of the oldest unprocessed message — is its real health metric. Depth tells you how much is queued; age tells you how stale the world is for a user (Queue Backlog).

The consume loop, and where each step fails

A consumer looks like a callback. It is actually a loop with six distinct failure points, and the reason consumers are hard is that five of them produce no error the author will ever see.

Walk the steps below and note which side of the ack each effect falls on. Everything before the ack can be repeated; everything after it can be lost. That single boundary determines every other design choice in this lesson.

One message, start to finish

1
Receive
Broker delivers the message with its envelope and delivery count.
fails by Delivered again after a rebalance or a lease expiry, while the previous attempt may still be running.
2
Parse and validate
Decodes the payload and checks the schema version.
fails by Throwing on an unknown field or an unknown type, turning a producer's additive change into an outage.
3
Dedupe check
Has this event id already been processed?
fails by Check and effect in different transactions, or a dedupe record that expired before redelivery.
4
Order check
Is this entity version newer than what is stored?
fails by Absent — an older event overwrites newer state and nothing reports it.
5
Effect
The actual work: write, call, enqueue.
fails by Partially applied across two systems; slow enough to exceed the broker's lease and trigger redelivery mid-flight.
6
Ack
Marks the message done so it is not redelivered.
fails by Before the effect, losing work permanently. After a crash, duplicating it — which is the survivable choice.

Nothing upstream is watching any of these. The producer got a 200 long ago.

The consumer failure table

These are the situations a consumer will actually meet, in roughly the order teams meet them. Each row is a decision to make before writing the handler rather than after the incident review.

What goes wrong in a consumer, and what to do about it

Trigger	Symptom	Cause	Response
Crash between effect and ack	The same fulfilment record created twice.	At-least-once delivery; the ack is not part of the effect's transaction.	Dedupe on event id, written in the same transaction as the effect (Job Idempotency).
Malformed or unhandleable payload	Retries forever; everything behind it stops.	Unbounded in-place retry of a permanent error.	Classify permanent vs transient; dead-letter permanent errors immediately (An Error Taxonomy That Maps Cause to Response).
Downstream dependency returning 500s	Consumer error rate spikes; dependency gets worse.	Tight retry loop adding load during the dependency's worst moment.	Backoff with jitter, a retry budget, and a breaker on the dependency (Circuit Breakers).
Two updates for one entity, two instances	The older value is stored. No error anywhere.	Concurrent processing without an ordering key or a version check.	Partition by entity id and reject events older than stored version.
Producer adds a new event type	Consumer throws on every message of that type.	Exhaustive switch with a default that raises.	Ignore unknown types; log and count them.
Consumer stopped an hour ago	No errors. Search is stale, emails are not sent, nobody notices.	Alerting on error rate rather than on progress.	Alert on oldest-unprocessed-message age per group; treat flat throughput with non-empty backlog as an outage.
Bug fixed, topic replayed	Downstream dependency melts under months of history at full rate.	Replay is a load test nobody scheduled.	Rate-limit the replay path and use a separate consumer group (Rate Limiting).

A handler that survives redelivery

DATABASE-SPECIFICRelies on a unique constraint and a multi-statement transaction. On a store without cross-document transactions the equivalent is one conditional write that carries the event id and the version together, so the claim and the effect are one operation.

The shape below is deliberately boring and contains the three checks that matter: the dedupe record and the effect share one transaction, the entity version is compared before applying, and unknown types are ignored rather than thrown.

Note what is absent. There is no attempt to detect whether this is "really" a duplicate by comparing payloads, and no attempt to prevent redelivery. Both are unwinnable; making repetition harmless is winnable.

Consuming OrderPlaced

1async function onOrderPlaced(msg: EventEnvelope) {
2  if (msg.type !== 'orders.OrderPlaced') return  // ignore, do not throw
3 
4  await db.transaction(async (tx) => {
5    // 1. dedupe: unique constraint on event_id does the work.
6    //    Insert first so a concurrent redelivery loses here, not later.
7    const claimed = await tx.processedEvents.insertIfAbsent({
8      eventId: msg.eventId,
9      consumer: 'fulfilment',
10    })
11    if (!claimed) return  // already handled; ack and move on
12 
13    // 2. ordering: never let an older event overwrite newer state
14    const current = await tx.fulfilment.findByOrderId(msg.orderId)
15    if (current && current.orderVersion >= msg.orderVersion) return
16 
17    // 3. effect, in the same transaction as the dedupe record
18    await tx.fulfilment.upsert({
19      orderId: msg.orderId,
20      orderVersion: msg.orderVersion,
21      status: 'pending',
22    })
23  })
24  // ack happens after the transaction commits.
25  // A crash before the ack redelivers; step 1 absorbs it.
26}

The dedupe insert comes before the effect and inside the same transaction. If it came after, two concurrent redeliveries could both pass the check and both apply the effect — the read-then-write gap is the bug this ordering closes.

How to build it

Most important first.

Dedupe on the event id, not on the payload contents. Record processed ids in the same transaction as the effect so the check and the write cannot disagree (Idempotency Storage).
Prefer effects that are naturally idempotent: an upsert keyed on the source id, a state transition guarded by the current state, a counter derived by recomputation rather than increment (Atomic Operations).
Publish everything about one entity with the same ordering key, and within the consumer compare the entity version before applying — reject anything older than what you have stored (Optimistic Concurrency).
Bound retries, back off with jitter, and dead-letter with enough context to replay by hand (Backoff and Jitter, Dead-Letter Queues).
Handle unknown event types and unknown fields by ignoring them, not by throwing. A producer adding a type must not break existing consumers.
Keep the consumer's work small and its transaction short. Long-running work inside a consumer is a job the consumer should be enqueueing, not doing (Background Jobs).
Propagate the correlation id from the event envelope into the consumer's logs, spans and outgoing calls (Request Context Propagation).

What can go wrong

Failure modes

Deduplication table with a shorter retention than the broker's redelivery window: an event redelivered after the dedupe entry expired is processed again (Idempotency Storage).
Dedupe check and effect in separate transactions, so a crash between them either duplicates the effect or permanently marks unprocessed work as done.
A consumer that acks on exception to "keep the queue moving", quietly discarding real work.
Ordering key chosen as the customer id when the invariant is per order — everything looks ordered until a customer places two orders at once.
A dead-letter queue nobody reads. The messages are safe, the capability is broken, and the graph is flat.
Consumer autoscaled on CPU while the bottleneck is a downstream dependency, so scaling up increases the load that caused the lag (Parallelism Moves the Load Downstream lives in Concurrency).

What can race

Two instances in the same group processing two events for one entity concurrently, so the write that started earlier commits later and stale data wins (Backend Races).
A redelivery overlapping the original attempt that has not actually died — two concurrent executions of the same handler for the same event id, which a dedupe check with a read-then-write gap will not catch (Duplicate Detection).
A rebalance moving a partition to another instance while the previous owner is mid-transaction, producing two writers for the same key for a short window.

Security

Event payloads are untrusted input. Validate them at the consumer boundary with the same rigour as an HTTP body — a consumer that trusts a field because "we publish it ourselves" trusts a different service's bugs (The Trust Boundary).
The consumer runs with its own credentials and no user context, so authorization cannot be inherited from a request. The tenant scope must come from the event and be enforced explicitly (Tenant Isolation).
Dead-letter queues accumulate full payloads, frequently containing personal data, and are usually the least access-controlled store in the system.

Misreads

"The broker guarantees exactly-once, so I do not need idempotency." Even where a broker offers exactly-once within its own boundary, your effects land in your database — the guarantee does not extend to the write your handler performs (At-Least-Once Delivery).
"Duplicates are rare, so I will handle them later." Duplicates arrive during deploys, rebalances and network blips, which are exactly the moments you are least able to investigate.
"Events are ordered." They are ordered within one partition. Two events published a millisecond apart on different keys have no defined order at all.
"Queue depth is my health metric." Depth zero with a stuck consumer looks identical to depth zero with a healthy one. Age distinguishes them.
"Consumers are simpler than handlers because there is no response." They are harder: no caller is watching, no user retries, and every failure is silent.

Operating it

How you see it in production

Per consumer group: messages processed, failures, retries, dead-lettered, and oldest unprocessed message age. Alert on age.
A duplicate-suppressed counter. Non-zero is healthy and expected; a sudden spike means an upstream retry loop or a redelivery storm.
Log the event id, type, version, attempt number and correlation id on every processing attempt — an event processed three times should be identifiable as one event (Structured Logging).
Track the delta between the event's occurredAt and processing time. That number, not queue depth, is what a user experiences as staleness.

What changes at 10x and 100x

Throughput scales with instances only up to the number of partitions or ordering keys. Beyond that, adding consumers adds idle consumers — the partition count is the real concurrency ceiling.
More instances means more concurrent writers to the same rows, so the per-entity ordering and version checks become load-bearing rather than theoretical.
At high volume, a per-message database round trip for deduplication is itself the bottleneck; batching acks and dedupe lookups is the usual answer, and it widens the crash window.
Replay of a large topic after a bug fix is a load event in its own right: the consumer processes months of history at full speed into dependencies sized for normal traffic.

What this costs

Idempotency costs a storage write and a lookup on every message, and the dedupe store becomes a dependency of every consumer.
Strict per-key ordering caps parallelism at the key count. You are choosing correctness over throughput, and the ceiling is real.
Dead-lettering keeps the pipeline moving at the cost of an out-of-band process — someone must own reading and replaying the queue, and that ownership decays.
Batching improves throughput and makes failure semantics worse: a batch that fails halfway is neither processed nor unprocessed.

Where this applies

Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.

GENERALAt-least-once, ack points, bounded retry and dead-lettering apply to every broker and job runner.
SIMPLIFIEDUses "partition" for the broker's ordering unit. Log-based brokers give ordered, replayable partitions with consumer-managed offsets; classic queue brokers give per-queue delivery with broker-managed acks and often no replay. Which of the designs here are available differs sharply between the two (Queue Semantics).
DATABASE-SPECIFICPutting the dedupe write and the effect in one transaction requires a store with multi-statement atomicity. Without it, the standard fallback is a conditional single-document write that carries both — different code, same invariant.

Where the depth lives

This domain teaches the application-side mechanism and hands the rest off.

API Designwebhook-consumer-idempotency retryability idempotency-vs-deduplication

Architecturemessage-queues kafka-partitions idempotency

Performancequeue-age queue-backlog worker-pool-saturation

Concurrencyproducer-consumer ordering-guarantees worker-pools

Domains that do not exist yet

— Distributed Systems — consumer-group rebalancing, offset management and why exactly-once processing is a property of the effect rather than of delivery.