Distributed Systems
How do I design a system that stays correct, available, understandable and recoverable when computation and data are spread across machines that talk over a network which loses, delays, reorders and duplicates their messages?
Everything here is a consequence of two sentences
A local call returns or throws. A remote call has a third outcome — no answer — and the syntax hides it. Every retry, every idempotency key, every saga and every timeout budget in this domain exists because of that third outcome.
The request may never have arrived, arrived and been dropped, executed and had its reply lost, or still be running. Three of those four mean the work did happen. Code that treats a timeout as a failure is wrong more often than it is right.
The rest of the domain is the work of getting back, in protocol and at a price, the four things a machine boundary takes away: shared memory, one clock, instant communication, and any reliable way to know who is alive.
Start from what you are actually seeing
Nobody arrives asking which consistency model they want. They arrive with a duplicate charge, a queue that will not drain, or two nodes that both think they are the leader. Pick the symptom and the answer is what to think about — never a product to install.
The outcome is unknown, not failed
Three of the five realities behind a timeout involve the work having completed. Treating it as failure is wrong more often than it is right.
- 1A Timeout Tells You Nothing About Whether It HappenedService A calls Service B and the call times out. The single most common mistake in distributed systems is treating that as "it failed". It is not a failure result — it is the absence of a result, and five different realities produce it.
- 2Partial Failure: The Founding ConditionA single machine fails all at once — the process is running or it is not. A distributed system fails in pieces, at different times, and each surviving piece has a different and incomplete picture of which pieces those were. Every other idea in this domain is a response to that.
- 3The Retry Is a Decision, Not a ReflexA timeout leaves the outcome unknown. Retrying and not retrying are both guesses, with different costs — and the moment you retry, the receiver has no way to tell your second attempt from a genuine second request unless you gave it one before the first attempt left.
The chain every lesson answers
Two of these fields are required and are what make this a domain rather than a pattern catalogue: the guarantee — the property claimed and its scope — and what a node can actually know, as opposed to what it is inferring.
Tools
Every module, every lesson, each one naming the guarantee it claims and how it fails.
Answer questions about your workload and land on a design — with what it actually promises, and explicitly what it does not.
A symptom, then what each party observed and concluded. Commit to a cause before the reveal; the reasonable reading is usually the wrong one.
Evidence you have to interpret, with the revealing item withheld until you commit.
What a strong answer contains, the green flags, the red flags, and a concrete failure to reason about.
Every simulator in the domain in one place: spacetime diagrams, quorum overlap, replicated logs, linearizability checks, partition drills.
A staged path through the domain, from the machine boundary to multi-region operations.
Symptom on the left, what to think about on the right — never symptom to product.
Learning modules
What a machine boundary actually changes
Partial failure, ambiguity, and what a node can know
Clocks lie; causality is what you can trust
Copies buy availability and cost you agreement
Naming the guarantee precisely enough to build on
Two writes, no order, one answer required
Agreement under failure, and its price
Atomicity across services you do not control
Retries are unavoidable; duplicates are the tax
Brokers, queues, logs and what each guarantees
Time is two different things at once
Splitting data without splitting correctness
When nodes must agree before acting — and when they need not
Who is in the cluster, and who is merely quiet
Bounded behaviour when demand exceeds capacity
Time budgets across a call graph
Copies of copies, and when they go stale
Layers beneath a distributed database
Moving work to data, and paying for the shuffle
Physics sets a floor on coordination
Detect, contain, recover, reconcile, verify
Where to cut, and what the cut costs
Agent workflows are distributed systems
What this domain is not
Not a glossary of CAP, Raft, Kafka and microservices, and not a product tour.
Architecture tells you which pattern to pick; Database tells you how replication works; Networking tells you how machines communicate. This domain tells you what the pattern actually guarantees, what the replication cost you in agreement, and what unreliable communication does to correctness. Where a lesson touches a mechanism that another domain owns — a saga, a consistent hash ring, a Kafka partition, an idempotency key — it links that lesson and moves straight on to the reasoning. Nothing here asks you to memorize a product; every lesson asks what guarantee is claimed, what each node can know, what fails, where agreement is unavoidable, and what it costs.
It is a system whose correctness depends on a component that fails independently and whose state you can only learn about through messages that may be lost, delayed, reordered or duplicated. One app plus one managed database already qualifies. Forty services that share a database and die as a unit are distributed in the worst way and coordinated in none of the good ways.
A guarantee names a property and its scope: *linearizable for single-key operations*, *read-your-writes for the session that wrote*, *at-least-once within a partition*. Strip the scope and you have marketing. Every lesson here is required to state one, which is why none of them can end at "it is consistent".
In an asynchronous network a crashed node and a slow node produce byte-for-byte identical evidence at every observer: silence. You do not detect failure, you decide to treat a node as failed, get agreement on the decision, and fence the old one so being wrong is survivable rather than catastrophic.
Concurrency has shared memory and shared fate: when the process dies, both threads die, and neither is left holding half a decision. Take away shared memory, a common clock, and any reliable oracle for "is the other party alive", and the reasoning changes shape rather than scale.
Every lesson carries required coordination, failureModes and alternatives fields. A design that cannot name what an operator sees when it breaks, or the simpler thing to try first, is a default being taught rather than a decision being made — and in this domain the price is paid in availability, not only in latency.
Simulated clusters, quorum arguments and consensus claims are labelled protocol, assumption, typical or simplified. "R + W > N means you read your write" is true by arithmetic and false under a sloppy quorum; the label is where that difference lives.