Event Keys and Partition Assignment
The key hashes to a partition, and the partition is the scope of ordering. Change the partition count and the hash re-maps, so a key's future loses order against its own past.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Which records need to stay in order relative to each other, and what key makes that true?
Any downstream that folds a sequence of changes into a state: a staging model taking the latest change per order, a stream processor keeping per-customer running totals, a materialised view of current inventory. Each of them needs all records for one entity to arrive in the order they happened, and none of them needs — or can have — a global order (Stateful Stream Processing).
The key defines a key group: all records sharing that key, guaranteed to sit in one partition and therefore in one order. That group is the real unit of ordering in the system, and choosing the key is choosing what "in order" means for this topic.
Publish without a key and let the client distribute records across partitions however it likes. Throughput is even, no partition runs hot, no thought is required, and for a stream of independent events — page views, log lines, sensor readings — it is exactly right.
The events turn out not to be independent. Two updates to order 7731 land in different partitions, are read by different consumer instances, and the staging model records placed as the latest state of a shipped order (CDC Ordering and Transaction Boundaries).
- The events turn out not to be independent. Two updates to order 7731 land in different partitions, are read by different consumer instances, and the staging model records
placedas the latest state of a shipped order (CDC Ordering and Transaction Boundaries). - The key is set to something too coarse — the country, the event type, the tenant in a system with one dominant tenant — and one partition receives most of the traffic while the rest idle (Data Skew).
- The key is set to something too fine, like the event id. Every record is its own key group, which is arithmetically identical to having no key at all as far as ordering is concerned, while looking like a deliberate choice.
- The partition count is raised for throughput. From that moment a key's new records may hash to a different partition than its old ones, so the entity's history is split across two independently-read logs and its ordering against itself is gone.
- The key includes a field that changes — a customer's current region, an order's status — so the same entity moves partitions mid-life and the sequence breaks with no configuration change at all.
What is actually happening
- A producer computes a hash of the key bytes and takes it modulo the partition count. Same key plus same partition count gives the same partition, always. That is the entire mechanism, and both halves of "same partition count" matter.
- With no key, records are distributed by the client — round-robin, or in sticky batches — so consecutive records for the same entity routinely land in different partitions. This is a distribution strategy, not an ordering one.
- Because the assignment is
hash(key) mod N, changing N changes the mapping for most keys. This is plain modular arithmetic rather than a broker quirk: it is the same reason naive hash-based sharding forces a full reshuffle when a shard is added, and the reason Consistent Hashing exists for systems that need to add capacity without re-mapping everything. - Nothing re-maps records already written. Old records stay in the partition they were written to, so after a partition count change an entity has records in two partitions, read concurrently by two consumers, with no relationship between their offsets (Topics and Partitions).
- Key distribution decides partition load directly. The broker does not balance traffic; it applies the hash you asked for. If ten per cent of your traffic carries one key, ten per cent of your traffic is on one partition and no partition count fixes it (Data Skew).
- Compacted topics use the key as the identity for retention: the latest record per key survives and earlier ones are discarded. On a compacted topic the key is therefore also a data-retention decision (Retention and Replay).
Key in, partition out
The assignment rule is two lines long and every consequence in this lesson follows from it. Hash the key, take it modulo the partition count, publish there. No key means no hash, so the client distributes however it likes.
What makes this worth writing out is the second argument. The mapping is not a function of the key alone — it is a function of the key and the current partition count. That second argument is stored nowhere in the record, appears in no schema, and is changed by people whose goal is throughput.
The practical reading: hash(key) mod N is a statement about ordering scope. Two records share an order if and only if they share a key and were published under the same N. Design as though N will eventually change, because on any long-lived topic it will.
1import zlib2 3# Stands in for the client's hash function — the specific algorithm differs4# by client, the modulo structure does not.5def partition_for(key: str, partitions: int) -> int:6 return zlib.crc32(key.encode()) % partitions7 8keys = ["C-1041", "C-2277", "C-3390", "C-4815", "C-5062", "C-6108"]9 10for k in keys:11 before = partition_for(k, 6)12 after = partition_for(k, 12)13 moved = "MOVED" if before != after else "stayed"14 print(f"{k} 6->p{before} 12->p{after} {moved}")15 16# C-1041 6->p3 12->p9 MOVED17# C-2277 6->p3 12->p3 stayed18# C-3390 6->p4 12->p4 stayed19# C-4815 6->p5 12->p11 MOVED20# C-5062 6->p4 12->p10 MOVED21# C-6108 6->p3 12->p3 stayed22#23# Nothing rewrites the records already in partitions 3, 4 and 5.24# C-1041's history is in p3; its future is in p9. Two logs, read by two25# consumers, with no ordering relationship between them.Doubling is the kindest case — a key stays put whenever its remainder was already below the old count — and it still moved half of these. A change from 6 to 8, or 6 to 10, has no such structure and moves nearly everything. The teaching is not the fraction; it is that the records already written are never re-mapped, so an entity ends up split across two partitions.
The partition-count trap, drawn out
The failure has a shape worth memorising because it is so easy to mistake for an application bug. A topic runs correctly for months with per-key ordering. Consumer lag grows. Someone raises the partition count, lag drops, and everyone moves on. Two weeks later a report of "order statuses going backwards" arrives, affecting a minority of orders, starting on a date nobody connects to the topic change.
The table below is the same six keys, before and after. What matters is the third column: for a moved key, its records are now split across two partitions that are read independently, so a new record can be processed before an older one. Only the moved keys are affected, which is why the symptom is partial and irreproducible.
And note what is *not* wrong. The producer is correct, the broker is correct, every record is durably stored once, in the right place per its mapping, and every consumer is reading its assigned partitions in order. There is no component to blame, which is precisely why this needs to be a design-time decision rather than a debugging skill.
Key by entity id, trust that all of an entity's records are in one partition, and have the consumer apply changes in the order it reads them. Correct, simple, and dependent on two conditions that live in the broker's configuration rather than in the data.
Key by entity id for locality *and* carry a monotonic value on each record — the source database's log position, a producer sequence number, a version. The consumer orders by that field and ignores arrival order entirely.
Partition assignment is a function of the partition count, which is operational configuration that changes for throughput reasons by people who are not thinking about ordering. A monotonic field on the record is immune to re-maps, to rebalances, to replays and to a consumer being restarted — it turns ordering from an infrastructure property into a data property, and only data properties survive an infrastructure change.
| Key | Partition at N=6 | Partition at N=12 | What happens to this entity |
|---|---|---|---|
| C-1041 | p3 | p9 | History in p3, future in p9. Two independently-read logs, so a later change can be applied before an earlier one. Ordering against its own past is gone. |
| C-2277 | p3 | p3 | Unaffected. Its records remain one contiguous ordered sequence, which is why the bug report covers only some customers. |
| C-3390 | p4 | p4 | Unaffected, for the same reason. Roughly half the keys survive a doubling and none of them tells you which half. |
| C-4815 | p5 | p11 | Split. p5 was busy and p11 is new and empty, so the new records are typically read *sooner* than the old ones — the reordering is systematic, not random. |
| C-5062 | p4 | p10 | Split, with the same bias toward the new partition being ahead. |
| C-6108 | p3 | p3 | Unaffected. Note that C-2277 and C-6108 still share a partition and still have no meaningful order relative to each other — sharing a partition is not sharing a key group. |
Key choice is also a skew decision
Every key choice has two consequences and teams usually only weigh one. The ordering consequence is deliberate — you chose the key to get per-entity order. The distribution consequence is inherited from the business, and it is the one that decides whether one consumer instance is permanently saturated.
Real key distributions have heads. One tenant is much larger than the rest, one product accounts for a third of orders, one region dominates. The hash does not know or care: it sends every record for that key to one partition, one consumer instance and one downstream task. The rest of the group is idle and the topic-level metrics say the group is under-utilised.
The available responses are unattractive in different ways, which is why choosing the key well the first time is worth real effort. You can accept the hot partition and size the whole group for it. You can salt the key — append a small random suffix — which restores distribution and destroys per-key ordering, so it is only available when the key was not there for ordering. Or you can split the topic, routing the dominant key to its own topic with its own consumer and its own scaling.
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Records per partition per hour, compared across partitions. | The key distribution is spreading traffic the way the design assumed. | Hot keys, a partition count change, a null-key regression that turned a keyed topic into a round-robin one. | Even distribution that is still wrong — many keys spread nicely across partitions can still each be in the wrong partition after a re-map, and the counts look perfect. |
| Per-key monotonicity of a version, source log position or state machine in the staging model. | Changes for one entity are being applied in the order they happened. | Null keys, a re-map after a partition count change, a mutable field used as the key, and a consumer that lost ordering during a rebalance. | Reorderings that preserve the checked field; and it cannot tell you *why* — a genuine late arrival and a partition re-map look identical here. |
| Distinct partitions observed per key over a rolling window, expected to be exactly one. | Each key group lives in one partition, as the design requires. | The partition-count trap directly, and it catches it as soon as the first re-mapped key publishes rather than when a report is wrong. | Keys with no traffic in the window; and it will flag a deliberate, correctly-managed re-key migration as a failure, so it needs a documented exception during a cutover. |
| Top keys by share of total records, alerted above a chosen fraction. | No single key can saturate one consumer instance. | A growing tenant before it becomes an incident, and a bug that collapses many keys into one default value. | Skew in payload *size* rather than record count — a key with few but enormous records saturates the same instance and this check says nothing (Data Skew). |
The third check is the one almost nobody has and the one that catches the trap this lesson is about. It is a group-by over a sample, it is cheap, and it fires on the day of the change instead of two weeks later.
How to build it
Most important first.
- Write the ordering requirement as a sentence before choosing the key: "all changes to one order must be applied in order". The key is whatever makes that sentence true, and if you cannot write the sentence you do not need a key.
- Key by the entity whose state you are reconstructing — order id for order changes, customer id for customer changes. Keying by something broader buys ordering you do not need and pays for it in skew.
- Use an immutable identifier. A key built from a mutable attribute silently moves an entity between partitions when that attribute changes, which is the same failure as a partition count change and much harder to spot.
- Do not rely on partition assignment alone for correctness. Carry a monotonic value — the source log position, a producer sequence, a version number — and have the downstream order by it. Then a re-map degrades performance rather than correctness (CDC Ordering and Transaction Boundaries).
- Check the key distribution before committing to it. Count records per key over a representative period; if the largest key is a significant share of the total, either accept a hot partition or design around it with salting and a downstream re-aggregation (Salting a Skewed Key).
- Treat partition count as part of the topic's contract and change it the way you would change a primary key: with a migration to a new topic and a cutover, not with a configuration edit (Contract Enforcement).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Records sharing a key are in one partition and therefore totally ordered — for as long as the partition count and the key are unchanged. Both conditions are invisible in the data and neither is enforced by anything.
- Records with different keys have no ordering relationship, even if they share a partition by hash collision. Sharing a partition gives them an arbitrary total order, not a meaningful one.
- No guarantee of even distribution. The hash spreads keys, not traffic, and traffic per key is a property of your business.
- No guarantee that a key group stays on one partition across a partition count change. This is the trap of the lesson and it is guaranteed to break rather than merely permitted to.
- On a compacted topic, the guarantee is only that the latest record per key survives. Any intermediate record may be gone, so a consumer that needs the sequence must not read a compacted topic (Kafka as a Log, Not a Queue).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Assert per-key monotonicity in staging: order each key's records by the monotonic field you carry and confirm that the state machine never goes backwards. This is the check that turns an invisible ordering break into a failed test.
- It misses reorderings that happen to preserve the checked field, and it cannot distinguish "records genuinely arrived out of order" from "this key changed partition" — same symptom, different fix. Pairing it with a per-key partition-count check closes that gap.
- Also track records per partition per hour as a distribution. A step change in shape is the signature of a partition count change or a key change, and it appears before anyone reports a wrong number (Volume Anomalies).
- A hot key makes freshness entity-specific: the entities on the busy partition are minutes or hours stale while everything else is current, and the topic-level lag average describes neither.
- Keying does not change end-to-end latency for a healthy topic. It changes which records are behind when something is behind, which is a more useful property than it sounds because it makes staleness attributable.
- Removing a key to improve distribution improves freshness for the previously-hot entities and destroys ordering for all of them. That trade is almost never worth taking without a downstream that reconstructs order from a record field.
- The key is part of the topic's contract even though it appears in no schema. Changing it re-maps every entity and breaks ordering against all published history (Breaking Schema Changes).
- The partition count is the second invisible half of that contract. A schema registry will happily approve a change to neither, because neither is in the schema (Schema Registry).
- Adding a field to the key — going from
customer_idtocustomer_id + regionto spread load — is a full re-map dressed as an optimisation, and it will be proposed during an incident about a hot partition, which is the worst possible moment to accept it.
- After a partition count change, per-key ordering across the boundary is not recoverable from the topic. It is recoverable from the records if — and only if — they carry a monotonic field, which is the argument for carrying one before you need it.
- The safe way to change partitioning is a new topic: publish to both, let consumers catch up on the new one from the earliest offset, validate the reconstructed state against the old, then cut over and retire the old topic (Replay from the Log).
- A replay after a re-map replays records from both mappings, so the interleaving on replay differs from the original run. A downstream that orders by a record field produces the same answer; one that relies on arrival produces a different one, and nothing reports the difference (Idempotent Data Pipelines).
- A hot partition cannot be rebalanced in place. Salting the key spreads it and requires the downstream to re-aggregate across salts, which is a transformation change as well as a topic change (Salting a Skewed Key).
What can go wrong
- Ordering lost silently at a partition count change, affecting only entities whose hash moved, only for records after the change.
- A key built on a mutable field, moving entities between partitions during normal operation with no change event anywhere.
- A hot key saturating one consumer instance while the group looks under-utilised in aggregate (Data Skew).
- A null key used unintentionally — a missing field, a serialisation default — turning a keyed topic into a round-robin one for a subset of records, which is the hardest variant to notice because most records are still ordered.
- A compacted topic keyed by an entity that also needs history, so intermediate states are discarded and a downstream that assumed it could replay the sequence finds only current state.
- Salting applied to fix skew without updating the downstream aggregation, producing per-salt partial results that look like a distribution change in the business (Salting a Skewed Key).
- "Keying guarantees ordering." Keying guarantees ordering *for a fixed partition count and a fixed key*. Neither condition is recorded anywhere in the data, and both get changed by people solving unrelated problems.
- "Adding partitions is a safe, reversible scaling operation." It is neither. It re-maps future records for most keys, and partition counts do not go down (Topics and Partitions).
- "Hashing distributes load evenly." Hashing distributes *keys* evenly. Load follows key traffic, and key traffic is never uniform (Hash Table).
- "We can key by anything that identifies the record." The key must identify the thing whose ordering matters. Keying by event id gives you a key per record, which is no ordering at all.
- "A hot partition means we need more partitions." A hot partition means one key is hot. Extra partitions do not split a key — only a different key or a salt does (Salting a Skewed Key).
- On a compacted topic the key decides what is retained indefinitely, so keying by a person's identifier creates an unbounded retention of their most recent record, which is a privacy decision made by a performance choice (Data Retention).
- Keys are visible in broker metrics, partition-level tooling and consumer logs far more often than payloads are. A key containing an email address or an account number leaks into operational surfaces that were never classified (PII in Pipelines, Secrets in Logs).
Operating it
- Records per partition per interval, as a distribution rather than a total. Skew and re-maps are both immediately visible here and nowhere else (The Backlog Arithmetic: Four Levers and a Drain Time).
- Top keys by volume over a rolling window, which turns "one partition is hot" into "this tenant is the reason".
- A per-key monotonicity violation counter from the staging model, which is the only place ordering breakage surfaces as data rather than as a support ticket (Data Quality).
- The topic's partition count itself, recorded as configuration history. It is the change most likely to explain an ordering incident and the least likely to be in any deployment log.
- At 10x, an acceptable key distribution usually stays acceptable — the shape of key traffic scales with the business rather than against it.
- At 100x, skew becomes the dominant concern. Uniform key distributions do not occur naturally, and the largest tenant, region or product tends to grow faster than the median, so the hot partition gets hotter (Data Skew).
- Key cardinality caps useful parallelism independently of partition count. Twenty distinct keys means at most twenty partitions can ever receive traffic, however many exist.
- Growth is what triggers the partition count change, which means the ordering trap is most likely to be sprung exactly when volume is rising and attention is elsewhere.
- Skew wastes consumer capacity directly: instances on quiet partitions are provisioned and idle while one instance is the bottleneck, so effective throughput is set by the busiest partition rather than by the total (Consumer Groups and the Parallelism Ceiling).
- Skew propagates downstream. The partition that receives most records produces the largest files and the largest state, and the task that processes it becomes the straggler that decides a batch job's runtime (Straggler Tasks).
- Salting costs a wider downstream aggregation — a second grouping pass to combine salts — permanently, in exchange for even distribution.
- A re-key migration costs dual publishing, a full replay of the new topic, and a validation pass. It is a project, not a change, which is the whole argument for choosing the key carefully once.
- Keying buys per-entity ordering and costs control over distribution. You get to choose one: even load, or ordered state per entity.
- A monotonic field on every record costs payload size and producer discipline, and buys ordering that survives re-maps, replays and rebalances. It is the cheapest insurance in this module.
- Salting buys even distribution and costs a permanent downstream re-aggregation plus the loss of per-key ordering — which means it is only available when ordering was not the reason for the key.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- BROKER-SPECIFICKafka hashes the key bytes and takes it modulo the partition count, so any count change re-maps most keys. Kinesis assigns by hash range and a reshard splits or merges ranges, so a change moves a contiguous slice of the key space rather than scattering it. Pub/Sub has no partitions and orders per ordering key, so the trap does not exist there and a different one — no ordering unless the key is set — does.
- SIMPLIFIEDThe code below uses a stdlib checksum to stand in for the client's hash function, which differs by client and by language binding. What transfers is the modulo structure and its consequence; the specific hash decides which keys move, never whether keys move.
- SCALE-SPECIFICOn a low-volume topic with one partition the key is irrelevant to ordering because everything is already ordered, and it matters only for compaction. Every concern in this lesson appears at the moment a second partition is added, which is usually a throughput decision made without reference to ordering at all.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns hash-based data placement in general — why modular hashing forces a reshuffle when capacity changes, and what consistent hashing and rendezvous hashing buy instead. This lesson is that same arithmetic showing up where people did not expect it.
- — Distributed Systems also owns what a monotonic per-entity sequence number actually costs a producer to generate, and why a global one is much harder than a per-key one.