Topics and Partitions
A topic is not one log — it is several. Ordering holds inside a partition and nowhere else, and that single sentence explains most of the surprises in a streaming platform.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
If a topic is ordered, why did the payment event arrive before the order event that caused it?
Anything that reconstructs state from a sequence of changes: a stream processor folding updates into a per-entity state store, a staging model picking the latest change per order, a materialised view. All of them are asking "what happened last?" and all of them get a wrong answer if they assume an ordering the topic never promised (CDC Ordering and Transaction Boundaries).
The unit of ordering, of parallelism and of storage placement is one partition. A topic is a naming and access-control boundary; a partition is the thing with physical and semantic properties. Every capacity, ordering or lag statement should name a partition, and one that names only a topic is usually hiding an average (Consumer Groups and the Parallelism Ceiling).
Create the topic with the default partition count, publish everything to it, and reason about it as a single ordered stream. This is exactly right for a topic with one partition, and one partition is a completely legitimate choice for a low-volume stream where ordering matters more than throughput.
The topic is given more partitions for throughput. Nothing else changes, no code is touched, and a consumer that folds updates into state now sometimes applies an older change after a newer one, because those two records went to different partitions and are read independently.
- The topic is given more partitions for throughput. Nothing else changes, no code is touched, and a consumer that folds updates into state now sometimes applies an older change after a newer one, because those two records went to different partitions and are read independently.
- A staging model picks "the latest row per order" by arrival order and starts producing statuses that go backwards — an order shows
placedafter it showedshipped, because the two change records were read from two partitions by two consumer instances (CDC Ordering and Transaction Boundaries). - Orders and payments are published to two topics and joined downstream on the assumption that a payment always follows its order. There is no ordering relationship whatsoever between two topics, so the join drops rows whenever the payment is read first (Stream Joins).
- A single tenant produces most of the traffic. Its records all hash to one partition, so one consumer instance falls steadily behind while the other eleven are idle and the topic-level lag average looks unremarkable (Data Skew).
- Someone sets the partition count to a very large number "for future scaling". Every consumer group now carries per-partition overhead, rebalances take longer, small batches produce many tiny files downstream, and the throughput was never the constraint (File Size and the Small-Files Problem).
What is actually happening
- A topic is a name. Underneath it are N partitions, each an independent append-only log with its own offset sequence starting at zero. Offset 500 in partition 0 and offset 500 in partition 3 are unrelated records with no temporal relationship at all.
- A producer chooses a partition per record — by hashing a key, by round-robin when there is no key, or explicitly. That choice is the only mechanism that puts two records in a knowable order, and it is made at publish time, not at read time (Event Keys and Partition Assignment).
- Partitions are distributed across brokers, so partition count is what lets a topic exceed the write and read capacity of a single machine. It is also the unit of replication: each partition has a leader and followers independently of the others (Partitioning and Sharding).
- Partition count is the ceiling on consumer-group parallelism, because a partition is assigned to at most one consumer within a group. Reading concurrency is therefore a decision made when the topic is created, not when the consumer is deployed (Consumer Groups and the Parallelism Ceiling).
- Consumers read partitions independently and in parallel, so records from different partitions interleave in whatever order the network and the scheduler produce. There is no merge step that restores a global order, because the information needed to do so was never recorded.
- Increasing the partition count is easy and one-directional: new partitions are added, existing records stay where they are, and future records for a given key may hash somewhere new. Decreasing it is not supported, which is the real reason to avoid an extravagant initial count (Event Keys and Partition Assignment).
A topic is N independent logs wearing one name
The diagram in most people's heads is a single line of records with a name on it. The real structure is N separate lines, each with its own offsets starting at zero, each living on a broker, each read by at most one consumer per group. The topic name is a routing and permissions concept and has no ordering meaning whatsoever.
Once that picture is in place, the surprises stop being surprising. Two records published a millisecond apart may be read minutes apart if they landed on different partitions and one of those partitions is behind. An entity whose records are spread across partitions has no reconstructable history. And the throughput you gained by adding partitions was paid for with exactly the ordering you gave up.
What follows from the picture is the design rule: ordering is something you buy with a key, and its scope is one partition. If you need "all changes to order 7731 in order", key by order id. If you need "every event in the system in order", you need one partition, and you should be sure you really need it.
| Stage | One row is | Breaks if |
|---|---|---|
| Topic | Nothing physical — a name, a permission boundary and a schema contract. | You reason about ordering, lag or availability at this level. All three are per-partition and a topic-level number is an average that hides the broken one. |
| Partition | One independent append-only log with its own offset sequence from zero. | You compare offsets across partitions. Offset 500 in two partitions are unrelated records, and the numbers being equal means nothing. |
| Record | One immutable fact at one offset in one partition. | You count records and call it entities. Three changes to one order are three records and one order (Grain: What Does One Row Represent?). |
| Key group | All records sharing a key — ordered, because they share a partition. | The partition count changes, or the key changes. Then a key's future records may land elsewhere and lose order against their own history (Event Keys and Partition Assignment). |
| Consumer assignment | One partition owned by exactly one instance in a group, for the duration of an assignment. | You assume it is stable. A rebalance moves it, and any in-memory state the old owner held is gone unless it was checkpointed (Checkpointing). |
Two of these rows — the topic and the key group — are the ones that carry no physical guarantee while feeling like they do. Both are where ordering bugs come from.
topic "orders" — one name, four independent logs
partition 0: 0 1 2 3 4 5 -> ordered, absolutely
partition 1: 0 1 2 -> ordered, absolutely
partition 2: 0 1 2 3 4 -> ordered, absolutely
partition 3: 0 1 -> ordered, absolutely
Between any two partitions: no order at all
Between offset 3 in p0 and
offset 3 in p2: unrelated records, no relationship
Consumer group "loader" (3 instances):
inst-A -> p0, p3
inst-B -> p1
inst-C -> p2
inst-D -> (nothing — 4 partitions, 4th instance idles)
What each instance sees: its partitions, in order.
What ANY instance sees: a global order. <- never trueThe ordering failures, and what each one actually is
Ordering problems are unusually hard to debug because the symptom is intermittent, entity-specific, and does not reproduce. The table below separates the four common causes, because they look identical from the dashboard and need entirely different fixes.
The distinction that matters most is between records that were never ordered (different partitions, different topics) and records that were ordered and got separated (a partition count change, a key change). The first is a design error and the fix is keying. The second is a change-management error and the fix is a migration, because the history published under the old mapping is now inconsistent with new records.
In all four cases the durable remedy is the same: do not rely on the transport for ordering. Carry a monotonic value on the record — the source database's log position, a producer sequence number, a version — and have the downstream reconstruct order from it. Then the partition layout becomes a performance decision instead of a correctness one (CDC Ordering and Transaction Boundaries).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Records for one entity published without a key, so they round-robin across partitions. | The entity's state flaps: shipped then placed then shipped again, for some entities, sometimes. | No key means no partition affinity, so consecutive records for one entity land in different logs and are read independently. | Key by the entity id so all of its records share a partition, and add a per-entity monotonicity assertion in staging so a regression is loud (Event Keys and Partition Assignment). |
| Partition count raised from 6 to 12 to increase consumer parallelism. | From the change onward, some entities show state going backwards. History before the change is fine, which makes it look like a recent code bug. | Hashing a key modulo the partition count re-maps most keys, so a key's new records land in a different partition from its old ones and the two are read in parallel. | Reconstruct order from a value on the record rather than from arrival; if strict per-key order is required across the change, migrate to a new topic and cut over rather than growing in place. |
| Orders and payments in separate topics, joined by "payment follows order". | The join intermittently drops payments, or attaches a payment to nothing, at a rate that varies with load. | There is no ordering relationship between two topics at all, and the two consumers progress independently. | Join on event time with an explicit lateness tolerance and a state store for unmatched records, or co-key both topics so related records share a partition (Stream Joins, Streaming State). |
| A consumer group rebalances during a deployment. | A short burst of duplicate or out-of-order processing around each deploy, invisible unless someone is looking. | Partition ownership moves between instances; the new owner resumes from the last committed offset, which may be behind what the old owner had processed but not committed. | Make the sink idempotent so redelivery is harmless, and checkpoint any in-memory state so a moved partition resumes rather than restarts (Idempotent Data Pipelines, Checkpointing). |
Choosing a partition count
There is no formula, because the inputs — peak consumer parallelism you will need, how skewed the keys are, how strong an ordering requirement you have, how much downstream file fragmentation you can tolerate — are all local. What there is, is a set of criteria and the knowledge that the number goes up and never down.
Start from consumer parallelism, because that is the constraint that bites first and the one that cannot be worked around later without an ordering consequence. Then sanity-check against skew: if one key is a meaningful fraction of the traffic, extra partitions do not help, because that key occupies exactly one of them however many there are.
Then apply the asymmetry. Overshooting costs overhead, rebalance time and small files — all annoying and all survivable. Undershooting costs a partition count change, which costs ordering for every key from that moment on. When in doubt, take the headroom.
What is actually constraining this topic — ordering, parallelism, skew, or nothing yet?
when Total ordering across the whole stream is a genuine requirement and the volume is comfortably handled by a single consumer.
cost No consumer parallelism at all, ever, and a single point of throughput. Perfectly correct until the volume grows, at which point the migration is the expensive kind.
when The normal case: per-key ordering is what you need, and consumers are or will be the bottleneck.
cost Per-partition overhead on brokers and in every consumer group, plus more, smaller files downstream. Cheap relative to changing the count later.
when Keys are few — a handful of tenants, a small set of regions — so partitions beyond that count can never receive traffic.
cost Parallelism is capped by the key space rather than by the partition count, so growth requires a different key rather than more partitions (Salting a Skewed Key).
when The topic is expected to grow substantially and per-key ordering across that growth is a hard requirement, so a later increase is unacceptable.
cost Overhead paid from day one, longer rebalances, and downstream file fragmentation that has to be managed with compaction (File Compaction).
How to build it
Most important first.
- Decide what must be ordered relative to what, and make that the key. "All changes to one order" and "all events for one customer" are ordering requirements; "all events in the system" is not achievable and asking for it means one partition (Event Keys and Partition Assignment).
- Size the partition count from consumer parallelism first and throughput second. It is the ceiling on how many instances can work on a topic, and raising it later has ordering consequences that raising a thread pool does not.
- Prefer a number you can live with for a long time, with headroom, over a number you expect to change. Every partition-count change is a re-map of the key space for future records (Event Keys and Partition Assignment).
- Never join two topics on assumed ordering. Join on event time with an explicit lateness tolerance, or key both topics so that related records share a partition and can be read in order by the same consumer (Stream Joins, Late Events).
- Reconstruct per-entity order downstream from a monotonic value carried on the record — a source log position, a version number, a producer sequence — rather than from arrival. Arrival order is an artefact of your infrastructure and it changes when you redeploy (CDC Ordering and Transaction Boundaries).
- Watch lag per partition, never only per topic. A single stuck partition is arithmetically invisible in a topic-level total and is one of the most common silent failures here.
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Total order within one partition, by offset. This holds absolutely and is the only ordering guarantee in the system.
- No ordering across partitions of the same topic. Not approximate, not usually-correct — none, and the timestamps do not rescue it because producers have separate clocks.
- No ordering across topics, for the same reason and more strongly.
- Records with the same key go to the same partition for a given partition count, so per-key ordering holds only as long as that count is unchanged (Event Keys and Partition Assignment).
- Each partition is replicated and durable independently. A partition can be unavailable — no leader — while the rest of the topic serves traffic normally, so "the topic is up" is not a well-formed statement.
- No guarantee of even distribution across partitions. Distribution follows the key distribution, and real key distributions are not uniform (Data Skew).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Assert monotonicity per entity in the staging model: for each key, the reconstructed sequence of a version, source log position or state field must never go backwards. This is the check that catches an ordering assumption that has quietly stopped holding.
- It misses reordering that happens to preserve the field being checked, and it cannot distinguish a genuine out-of-order arrival from a partition-count change that re-mapped a key — the symptom is identical and the causes need different fixes.
- Also check per-partition record counts against each other. A distribution that changes shape overnight usually means a key change or a partition count change rather than a change in the business (Volume Anomalies).
- Freshness is per-partition, because progress is per-partition. One partition blocked on a poison record has unbounded staleness while every other partition is current, and the topic average will describe neither situation.
- Adding partitions can improve freshness by raising the consumer parallelism ceiling — but only when consumers were the bottleneck. If a single consumer instance is slow, more partitions redistribute the same total capacity and change nothing (More Threads Is Not More Speed).
- A downstream that requires per-entity ordering must wait for the partition holding that entity, so a slow partition is a freshness problem for exactly the entities on it and for nobody else. That is invisible in every aggregate metric.
- The partition count is part of the topic's contract with anything that relies on per-key ordering, even though it appears in no schema. Changing it is a breaking change for those consumers and no schema registry will tell you (Breaking Schema Changes).
- Adding an event type to an existing topic changes what one record means for every consumer, including ones that filter on type and will now silently skip more. Whether to use one topic per event type or one per entity is a contract decision, not a naming preference (Naming Events).
- Changing the key changes the partition assignment for all future records, which breaks ordering against every record already published under the old key. This is the same failure as a partition count change and it is easier to do by accident (Event Keys and Partition Assignment).
- Replay is per-partition and independent: resetting a consumer group to an earlier offset means every partition is re-read from its own earlier position, and their interleaving on the replay will differ from the original run.
- That difference is only safe if the downstream is order-insensitive or reconstructs order from a field on the record. A pipeline that was accidentally correct because of arrival order will produce a different answer on replay, and there is no warning (Replay from the Log).
- A permanently stuck partition is recovered by publishing the bad record to a quarantine topic and committing past it, which is a deliberate data-loss decision that should be recorded rather than a routine unblocking action (A Dead-Letter Queue Is a Workflow, Not a Bin).
- Rebalancing a skewed topic cannot be done in place. The workable path is a new topic with a better key, dual publishing, and a cutover — the same shape as a table migration (Event Keys and Partition Assignment).
What can go wrong
- A consumer assuming global ordering and producing state that goes backwards for some entities, some of the time, in a way that is very hard to reproduce.
- One partition stuck behind a poison record while the topic-level lag looks merely elevated.
- Key skew concentrating most traffic on one partition, so one consumer instance is saturated and adding instances does nothing (Data Skew, Salting a Skewed Key).
- A partition count increase applied for throughput that silently breaks per-key ordering from that moment onward, with the previous history unaffected and therefore inconsistent with the new records (Event Keys and Partition Assignment).
- A partition count so high that downstream writes produce many tiny files, converting a broker decision into a query-performance problem in the lake (File Size and the Small-Files Problem).
- A partition without a leader after a broker failure, making part of a topic unavailable while monitoring that looks at the topic as a whole reports it healthy.
- "The topic is ordered." A partition is ordered. A topic with twelve partitions has twelve orderings and no relationship between them (The Event Log).
- "Timestamps let us re-sort into the true order." Producer clocks differ, and the true order is not recoverable from records that were never in one sequence. Event time lets you *window* correctly, which is a different and achievable goal (Event Time, Watermarks).
- "More partitions is always safer." More partitions costs overhead, longer rebalances and smaller downstream files, and reduces the ordering you have. There is no free headroom here.
- "Partitions balance load." Partitions balance *keys*. If the keys are skewed, the load is skewed, and the broker will do exactly what you asked (Data Skew).
- "We can lower the partition count if we overshoot." Partition counts increase and do not decrease. Overshooting is permanent for that topic.
Operating it
- Lag per partition per consumer group, plotted individually rather than summed. The maximum across partitions is the number worth alerting on (The Backlog Arithmetic: Four Levers and a Drain Time).
- Record count and byte rate per partition, which makes key skew immediately visible as a distribution rather than requiring anyone to go looking for it.
- Under-replicated and leaderless partition counts, which are the partition-level availability signals a topic-level view hides.
- A per-entity monotonicity violation counter emitted by the staging model, which is the only place an ordering assumption failing becomes visible as data rather than as a mystery (Data Quality).
- At 10x volume, partition count usually needs to rise — and doing so is a one-way door with ordering consequences, so the time to think about it is before it is urgent.
- At 100x, skew dominates. Uniform key distributions do not exist at scale, and the design question shifts from "how many partitions" to "what key makes the distribution survivable" (Event Keys and Partition Assignment, Salting a Skewed Key).
- Consumer group count multiplies per-partition overhead. Two hundred partitions and thirty groups is six thousand independently-tracked positions, which is a coordination workload in its own right.
- Below any scale at all, one partition is a perfectly good answer and gives you total ordering for free. Reaching for twelve partitions on a topic that carries a few records a second buys nothing and costs ordering.
- Fixed per-partition overhead on the brokers — file handles, replication streams, metadata — paid whether or not the partition carries traffic.
- Per-partition overhead in every consumer group: more fetch requests, more rebalance work, more state to track. This scales with partitions times groups, which grows faster than people expect.
- Downstream file count. Each partition's consumer typically writes its own files, so partition count multiplies the small-files problem in object storage and therefore the cost of every query that lists and opens them (File Compaction).
- Skew wastes provisioned consumer capacity: instances assigned to quiet partitions are paid for and idle while one instance is the bottleneck (Data Skew).
- Partitions trade ordering for parallelism, and the exchange rate is exact: one partition gives total order and no parallelism; N partitions give N-way parallelism and order only within each.
- A high partition count buys headroom and costs per-partition overhead everywhere, longer rebalances, and more, smaller files downstream.
- Keying for ordering buys per-entity correctness and costs distribution control, because real keys are skewed and the skew lands on one consumer.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- BROKER-SPECIFICKafka orders per partition and lets a consumer group hold one consumer per partition. Kinesis is the same idea named shards, but resharding splits and merges ranges rather than re-hashing, so the ordering break has a different shape. Pub/Sub provides no ordering at all unless an ordering key is set, and then it orders per key rather than per partition — three different answers to the same question.
- GENERALThe trade itself — ordering scope shrinks exactly as parallelism grows — is a property of partitioned systems generally, and shows up identically in sharded databases and in partitioned batch compute. What varies is whether the system lets you change the partition count and what happens to existing data when you do.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why a global order across independent machines is expensive rather than merely unavailable — total order broadcast, logical clocks, and what a timestamp from another machine is worth. This lesson is the applied consequence.
- — Distributed Systems also owns the group membership and rebalancing protocol that decides who owns which partition, and what happens to in-flight work when that decision changes.