Event Logs & Brokers
The durable, partitioned, replayable append-only log as data infrastructure. Topics, partitions, keys, consumer groups, offsets and retention — and why replay is the feature that matters most here.
An append-only, immutable, ordered sequence of facts that each reader moves through at its own pace — the primitive underneath brokers, replication, CDC and stream processing.
Two different products wearing one word. A queue distributes work and forgets; a log stores records and lets anyone re-read them. Neither is the upgrade of the other.
A partitioned, durable, append-only log with independent consumer groups and time-based retention. Records survive being read, which is the property everything else in a data platform is built on.
A topic is not one log — it is several. Ordering holds inside a partition and nowhere else, and that single sentence explains most of the surprises in a streaming platform.
The key hashes to a partition, and the partition is the scope of ordering. Change the partition count and the hash re-maps, so a key's future loses order against its own past.
A group divides a topic's partitions among its instances, one partition to at most one instance. Partition count is therefore the hard ceiling on parallelism, and instances beyond it do nothing at all.
Retention is not a storage setting. It is the maximum age of a bug you can fix by replaying instead of reconstructing — a recovery-window decision that happens to be paid for in disk.
Commit before processing and you get at-most-once. Commit after and you get at-least-once. There is no third option unless the commit and the output write share a transaction.