The question this answers
Which ordering does this queue actually guarantee, which one does my code assume, and what would the stronger one cost?
An event pipeline where producers emit account updates onto a queue and a pool of consumers applies them to a store — with two updates to the same account arriving close together.
The queue itself and the store the consumers write to. The ordering guarantee is a property of the path between them, and it is the thing that decides whether two updates to one account can be applied backwards.
For any single account, updates are applied in the order the producer emitted them, so the final stored value is the one the producer emitted last. Across different accounts, no ordering is required or promised.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
Four levels, and what each one actually promises
Ordering is not a single property that a system either has or does not. It is a ladder, and the useful skill is knowing which rung you are standing on. No ordering: messages may be observed in any order at all, including a later one before an earlier one from the same producer. FIFO per producer: everything one producer sent is observed in the order it sent it, with no promise about how it interleaves with other producers. Causal: if A happened before B in a way the system can see, every observer sees A before B; concurrent events may be seen in either order. Total: every observer sees every event in the same single order.
The prices rise steeply and unevenly. FIFO per producer is nearly free — a single connection or a single partition provides it structurally. Total order is expensive because it requires a single point that everything passes through, or an agreement protocol, and it removes the parallelism that made the system fast. Causal order sits between: cheaper than total, sufficient for most application invariants, and considerably more machinery than FIFO.
The bug pattern is always the same. The queue documents FIFO per producer. The consumer pool is scaled to eight. The code assumes total order because a single consumer used to provide it incidentally. Nothing in the type system, the API or the tests records the assumption, and it breaks the day someone changes a configuration value from 1 to 8.
- The default for anything parallel is "none" unless something specific is providing more.
- One consumer provides total order incidentally, which is why scaling the consumer pool is where the assumption breaks.
- Idempotent and commutative operations need no ordering at all, which is the cheapest way out of this whole conversation.
| Level | Promise | Cost | Typically provided by | Enough for |
|---|---|---|---|---|
| None | Any observation order at all | Free; maximum parallelism | Fan-out to many consumers, retries, multiple paths | Idempotent, commutative operations (counters, sets) |
| FIFO per producer | One producer's messages in send order | Cheap — one path per producer | A single connection; one partition; one queue consumer | Per-entity updates, if the entity maps to one producer |
| Per-key / partitioned | All messages for one key in order | Cheap; parallelism bounded by key count | Hash-partitioned log; consistent routing | Most application invariants — the practical sweet spot |
| Causal | If A caused B, everyone sees A first | Moderate — dependency metadata to track and check | Version vectors, dependency tracking | Comment-after-post, read-your-writes |
| Total | One order, identical for every observer | Expensive — a single serialization point | One serializing node; a consensus protocol | Ledgers, sequence numbers, leader election |
Where FIFO stops: two consumers, one producer
The queue below genuinely provides FIFO per producer: the producer's two events are *enqueued* in order and *delivered* in order. The guarantee ends at delivery. Two consumers take them at nearly the same moment, consumer B finishes first, and the store ends up with the earlier value. The queue kept its promise exactly; the application needed a promise about *application* order that nobody made.
This is the most important thing to internalize about ordering: the guarantee attaches to a specific boundary, and processing after that boundary is concurrent again. FIFO delivery to a pool of N consumers gives you nothing about the order effects land in the store, and it is precisely the setup that a single consumer made look correct during development.
The remedy is not a lock around the store, though that is the usual first attempt and it is worse than it looks: it serializes every account against every other account, converting a scalable pipeline into a serial one to protect an invariant that only concerns one account at a time. The right shape is in the next section — route by key so the ordering constraint is enforced by the partitioning rather than by mutual exclusion.
| # | Producer | Queue (FIFO per producer) | Consumer 1 | Consumer 2 | State |
|---|---|---|---|---|---|
| 1 | emit e1: account 42 -> 100 | · | · | · | queue=[e1] |
| 2 | emit e2: account 42 -> 250 | · | · | · | queue=[e1, e2] |
| 3 | · | deliver e1 to Consumer 1 | · | · | queue=[e2] C1 holds=e1 |
| 4 | · | deliver e2 to Consumer 2 — still in order at this boundary | · | · | queue=[] C2 holds=e2 |
| 5 | · | · | begins applying e1: fetches the account record (cache miss, slow) | · | db[42]=null |
| 6 | · | · | · | applies e2: db[42] = 250 | db[42]=250 |
| 7 | · | · | applies e1: db[42] = 100 | · | db[42]=100 ✕ The producer emitted 250 last, so the stored value must be 250. An older update overwrote a newer one. |
Buy ordering per key, not globally
The scalable answer is to make the ordering constraint match the invariant. The invariant is per account, so partition by account: hash the key, route every event for that key to the same partition, and give each partition exactly one consumer. Events for account 42 are then processed strictly in order by construction, while accounts 7 and 91 proceed fully in parallel on other partitions. Ordering is bought exactly where it is needed and nowhere else — this is why partitioned logs are shaped the way they are (Kafka-Style Logs: Topics, Partitions, Offsets in Architecture is the canonical implementation).
The costs are real and worth naming. Parallelism is now bounded by partition count, not by machine size. A hot key concentrates load on one partition and one consumer, and no amount of scaling helps because splitting it would break the guarantee you bought. Repartitioning to add capacity is disruptive because keys move between partitions and in-flight ordering spans the change. And per-partition head-of-line blocking is now possible: one poisoned event stalls every key in its partition, not just its own.
The alternative worth trying first, always, is to need less ordering. An operation that is idempotent and commutative — set-if-newer with a version, increment-by-delta, add-to-set — is order-independent, and then no ordering guarantee is required at any level. Attaching a version number to each event and rejecting stale ones (Optimistic Concurrency Control) turns the schedule above into a non-event: consumer 1 tries to apply version 1 over version 2 and is refused. That is usually cheaper than buying ordering, and it survives retries and duplicate delivery as well, which ordering alone does not.
- Ordering guarantees should be scoped to the invariant: per key, not per system.
- Parallelism is then bounded by partition count, and a hot key is a bottleneck you cannot scale out of.
- Cheaper still: make the operation idempotent and commutative, and need no ordering at all.
Key points
- Ordering is a ladder — none, FIFO per producer, per-key, causal, total — and each rung costs more parallelism than the one below.
- A guarantee attaches to a boundary. FIFO *delivery* to N consumers says nothing about the order effects are *applied*.
- A single consumer provides total order incidentally, which is why the assumption breaks when the pool is scaled.
- Partitioning by key buys ordering exactly where the invariant lives and keeps parallelism everywhere else.
- The costs of per-key ordering are hot keys, parallelism bounded by partition count, disruptive repartitioning and per-partition head-of-line blocking.
- Cheapest of all: make operations idempotent and commutative so no ordering guarantee is needed.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Identify the invariant that needs ordering and the scope it applies to — almost always per entity, rarely global.
- • Choose a key that matches that scope and ensure every message carries it.
- • Route deterministically: the same key always reaches the same partition, queue or actor.
- • Give each partition exactly one consumer, so within a partition processing is sequential and ordering is structural rather than enforced.
- • For anything crossing partitions, either accept no ordering or attach version numbers and make the operation reject stale updates.
- • FIFO delivery, concurrent application: e1 delivered before e2, but C2 applies e2 first and C1 then overwrites it with e1 — the older value wins and the queue kept every promise it made.
- • Partitioned: e1 and e2 both carry key 42, both go to partition 0, one consumer applies e1 then e2. No interleaving exists to break the invariant.
- • Cross-key, deliberately unordered: e1 for account 42 and e3 for account 7 are applied in either order on different partitions. Nothing is violated because nothing was promised.
- • Retry-induced reordering: C1 fails applying e1, retries after a delay, and applies it after e2 — retries defeat ordering even within a partition unless the operation checks a version.
- • Rebalance: partition 0 moves to a new consumer while e2 is in flight; the new consumer starts from the committed offset and may deliver e2 again, so ordering plus at-least-once delivery still needs idempotence.
- • Total order attempt: a single global consumer applies everything in one sequence. The invariant holds for every key, and throughput is now one consumer's worth for the entire system.
- • FIFO per producer guarantees one producer's messages are delivered in send order. It does NOT constrain interleaving with other producers, and does NOT constrain processing order after delivery.
- • Per-key partitioning guarantees all messages for one key are processed in order, provided each partition has exactly one active consumer.
- • It does NOT guarantee ordering across keys, and does NOT survive a consumer that processes messages from its partition concurrently — a common and silently fatal optimization.
- • Total order guarantees every observer sees the same sequence, at the cost of a single serialization point that is also a throughput ceiling and a failure domain.
- • No ordering level guarantees exactly-once processing. Retries and rebalances can redeliver, so ordering and idempotence are separate properties you need separately.
- • Nothing guarantees ordering across a retry boundary: a failed-and-retried message arrives after messages that came behind it, whatever the queue promises.
- • Total order concentrates all traffic through one serialization point, which becomes both the throughput ceiling and the contention point for the whole system.
- • Per-key ordering bounds parallelism at the partition count; a hot key means one consumer is saturated while others idle, and it cannot be split without losing the guarantee.
- • Head-of-line blocking within a partition: one slow or poisoned message delays every other key sharing that partition.
- • A store-wide lock used to recover ordering serializes unrelated keys against each other — the naive fix, and a much larger contention cost than the partitioning it replaces.
- • Lost update from out-of-order application: an older value overwrites a newer one, with no error and no message loss.
- • A pipeline correct at one consumer and wrong at eight, where the breaking change was a configuration value.
- • Hot-key saturation that cannot be scaled out, because splitting the key would break the ordering it was partitioned for.
- • Head-of-line blocking stalling a partition behind one bad message.
- • Reordering introduced by retries, defeating a partition-level guarantee that was otherwise correct.
- • Duplicate application after a rebalance, because ordering was bought but idempotence was not.
- • A consumer that "safely" processes its partition concurrently for throughput, silently discarding the guarantee the partitioning existed to provide.
- • Whenever an invariant is per entity: account balances, document versions, workflow state machines, per-user session updates.
- • When the ordering can be scoped to a key, so parallelism is preserved everywhere the invariant does not reach.
- • When making the assumption explicit is itself the win — writing "this consumer requires per-key ordering" prevents the pool from being scaled into a bug.
- • When total order is chosen for safety and becomes the system's throughput ceiling and single point of failure.
- • When the key is chosen badly, producing hot partitions that cannot be relieved.
- • When ordering is bought for operations that are already commutative, paying for a guarantee that was never needed.
- • When ordering is treated as sufficient for correctness, without idempotence — retries and redelivery will break it regardless.
- • Read the queue or transport documentation and write down which rung it actually provides. Most ordering incidents begin with nobody having done this.
- • Instrument out-of-order application directly: attach a producer sequence number per key and count how often a consumer sees a lower one than it last applied.
- • Per-partition lag and throughput skew, which exposes hot keys immediately.
- • Consumer count per partition, alarmed on greater than one — the configuration that silently voids the guarantee.
- • Retry and redelivery rates per key, since both reorder within an otherwise correct partition.
- • The key choice becomes a long-lived structural decision: it determines parallelism, hot spots and how repartitioning will go.
- • Repartitioning is genuinely hard, because keys move between partitions and ordering spans the transition.
- • Every consumer must be documented and tested as "one active consumer per partition, sequential within it", and that constraint is easy to violate with an innocuous concurrency optimization.
- • Idempotence must be implemented as well as ordering, because delivery guarantees and ordering guarantees are independent.
- • Cross-key invariants have no home in this model and force either total order or a different design entirely.
- • Make operations commutative and idempotent — set-if-newer with a version, increment-by-delta, add-to-set — and need no ordering guarantee at all. Try this first.
- • Attach a version and reject stale updates at the store (Optimistic Concurrency Control); this survives reordering, retries and duplicates together.
- • Use a database transaction with the appropriate isolation level, when the ordering concern is really a concurrent-update concern on a row.
- • Route the entity to a single owner — an actor, a lease, a session — so ordering is a consequence of exclusive ownership (The Actor Model).
- • Accept unordered processing and reconcile afterwards, when the invariant can be restored by a later pass.
Producers, a bounded queue, consumers
The producer is faster than the consumer
The lost update, step by step
| # | Task A — counter++ | Task B — counter++ | State |
|---|---|---|---|
| 1 | rA ← counter | · | counter=0 rA=0 rB=— |
| 2 | · | rB ← counter | counter=0 rA=0 rB=0 |
| 3 | rA ← rA + 1 | · | counter=0 rA=1 rB=0 |
| 4 | counter ← rA | · | counter=1 rA=1 rB=0 |
| 5 | · | rB ← rB + 1 | counter=1 rA=1 rB=1 |
| 6 | · | counter ← rB | counter=1 rA=1 rB=1 ✕ 2 increments completed, counter = 1 |
What people believe, and what is true
The queue is FIFO, so my events are processed in order.
FIFO describes delivery. With N consumers, processing order is completion order, and an older event can land after a newer one.
It has always worked, so the ordering is guaranteed.
It worked because there was one consumer. That is total order provided incidentally, and it disappears the moment the pool is scaled.
Total order is the safe default.
It is a single serialization point: the throughput ceiling and a failure domain for the whole system. Scope ordering to the key the invariant is about.
Per-key ordering means I do not need idempotence.
Ordering and delivery guarantees are independent. Retries and rebalances redeliver, and a redelivered message arrives out of order relative to what came after it.