The question this answers
When a node joins or leaves, how much data actually moves — and what happens to requests issued during the change?
Adding or removing one node relocates only the keys mapped to that node’s ring positions: K/N keys in expectation for K keys and N nodes. Every other key keeps the same owner. This is a guarantee about the *mapping*, not about balance, and not about what happens to requests during the transfer — both need separate mechanisms.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A router knows the ring it last received: a set of node positions. It does not know whether that ring is current. During a membership change, two routers holding different rings compute different owners for the same key, and each is internally consistent and confident. Nothing local distinguishes "I have the current ring" from "I have last minute’s ring".
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The construction, and why it is stable
Hash the key space onto a circle — conventionally the 128-bit output of a hash treated modulo 2^128. Hash each *node identifier* onto the same circle. A key belongs to the first node encountered going clockwise from the key’s position.
The crucial structural difference from hash(key) % N: the node count does not appear in the key’s mapping. A key’s position on the ring is fixed forever by its hash. What changes when membership changes is only *which node is next clockwise*, and that changes only for keys sitting in the arc that the joining or leaving node covers.
Remove a node and its arc merges into the next node clockwise — that node inherits the keys, everyone else is untouched. Add a node and it takes over the portion of its clockwise successor’s arc that now falls behind it — one node gives up a slice, everyone else is untouched. At most two nodes are involved in any single membership change, which is what makes the operation plannable.
Architecture covers the pattern itself and its lineage; this lesson takes the operational half. See the cross-domain link at the foot of the page.
1// positions: sorted array of hash(nodeId) values around the circle2function ownerOf(key: string, positions: number[], owners: string[]): string {3 const h = hash128(key)4 let lo = 0, hi = positions.length5 while (lo < hi) { // first position >= h6 const mid = (lo + hi) >> 17 if (positions[mid] < h) lo = mid + 18 else hi = mid9 }10 return owners[lo % positions.length] // wrap around the circle11}12 13// Note what is absent: the number of nodes never touches the key's hash.14// N appears only in the *search*, never in the *mapping*.How much data actually moves
The K/N figure is an expectation, and the gap between the expectation and reality matters operationally.
Removing one node from a ten-node cluster moves that node’s share — nominally 10% of the data — and it all moves to one destination: its clockwise successor. That successor now serves roughly twice its previous load while absorbing a full node’s worth of data over the network. A cluster that was at 60% capacity now has one node at 120%. This is the single most important practical consequence of the plain ring, and it is why real systems never deploy it without Virtual Nodes: Many Positions per Machine, and Why It Is Not Optional.
Adding one node to a ten-node cluster is gentler: the new node pulls roughly 1/11 of the data, all from one source. The source is briefly saturated; nobody else notices.
Compare the totals honestly against the modulo scheme: at N=10, % N moves ~91% of the data spread evenly across all senders and receivers; the ring moves ~9% concentrated on one pair. Which is worse depends on what you are protecting. The ring is far better for total bytes and far worse for peak per-node impact — and peak per-node impact is what causes incidents.
| Scheme | Fraction of data moved | Nodes involved in the move | Load the survivors inherit |
|---|---|---|---|
| `hash(key) % N`protocol | ≈ 90% | All ten | Even — 1/9 extra each, after a full reshuffle |
| Plain ring (1 position/node)protocol | ≈ 10% | Two: the departing node’s successor, plus the source | All on one node — it doubles |
| Ring + 256 virtual nodesassumption | ≈ 10% | All nine survivors | Even — ≈ 1/9 extra each |
| Fixed partitions + maptypical | ≈ 10% | As many as you choose to spread the reassignment over | Whatever the assignment algorithm decides |
The window where two nodes both believe they own the key
This is the part the pattern description omits, and it is where the bugs live. A ring change is not atomic across the fleet. There is a real interval — seconds to minutes, depending on how the ring is distributed — during which some routers use the old ring and some the new one, and during which the data itself has not finished moving.
Four distinct hazards live in that interval:
A read to the new owner before the transfer completes returns "not found" for a key that exists. To an application this is indistinguishable from a deletion, and it will happily act on it — write a default, create a duplicate record, or report the item missing to a user.
A write to the old owner after ownership moved lands on a node that will shortly stop serving that key. If the transfer already copied that range, the write is silently dropped when the old node relinquishes it.
Split writes: one client writes to the old owner, another to the new. Two live values for one key, and no version relationship between them, so even a merge rule has nothing to work with.
Broken read-your-writes: the same client writes via a router with the new ring and reads via a connection pinned to a node with the old ring, and does not see its own write (Read-After-Write: Letting a User See Their Own Change).
The mitigations, in increasing order of strength: have the *new* owner forward to the old owner for any key not yet transferred (cheap, hides the window for reads); have the *old* owner reject with an explicit "not owner, refresh your ring" rather than serving or 404ing (essential — this is the same discipline as in Range Partitioning: Scans You Keep, Hotspots You Inherit); stamp the ring with a monotonically increasing epoch and refuse requests carrying an older epoch than the node’s own (Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely, Terms and Epochs: Making Stale Leaders Harmless); or, strongest, route ownership changes through a consensus-backed membership so that ownership handover is a totally ordered event (Cluster Membership: A Belief, Not a Fact).
What the ring does not give you
It does not balance load. With N nodes at random positions on a circle, the arcs are not equal — they follow the spacings of N uniform random points, and the largest is far bigger than the average. That is the subject of Virtual Nodes: Many Positions per Machine, and Why It Is Not Optional and it is not a minor refinement; a plain ring is unusable in production for this reason alone.
It does not decide replication. The usual convention is that a key’s replicas are the next R *distinct physical* nodes clockwise. "Distinct physical" matters: with virtual nodes, the next three positions may all belong to the same machine, which would give you replication factor one while reporting three. Correct implementations walk the ring skipping repeats, and rack-aware ones skip same-rack nodes too (Fault Domains: What Fails Together).
It does not remove the need for a membership protocol. Something must tell every router that the ring changed, and that something is Gossip: Epidemic Spread Instead of Everyone Telling Everyone or a consensus-backed registry. The ring is the mapping function; membership is a separate, harder problem.
It does not survive an inconsistent node-id-to-position function. The positions are hash(nodeId); if two implementations hash node ids differently — a hostname versus an IP, with or without a port — they build different rings from the same membership list and route the same key to different nodes.
Key points
- The node count does not appear in a key’s mapping, which is why the mapping survives membership changes.
- One membership change involves at most two nodes and moves ≈ K/N keys — but on a plain ring it dumps all of it on one successor.
- The change is not atomic across the fleet: there is always a window with two rings in circulation.
- In that window, reads can report existing keys as missing and writes can be accepted then discarded — both silently.
- The safe discipline is refuse-and-redirect, never serve-or-404, plus an epoch so a stale ring is detectable rather than merely wrong.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Hash node identifiers onto a circular key space and keep the positions sorted.
- • Hash each key onto the same space; the owner is the first node position clockwise from it.
- • Replicas are the next R distinct physical nodes clockwise, skipping repeated physical nodes and, ideally, repeated fault domains.
- • On join: the new node takes the arc between its position and its counter-clockwise predecessor, streaming that range from its clockwise successor.
- • On leave: the departing node’s arc merges into its clockwise successor, which streams the range in.
- • Throughout the transfer, requests for keys in the moving arc must be forwarded to whichever node currently holds the data, not answered locally.
- • When the transfer completes, publish a new ring epoch; nodes reject requests carrying an older epoch for the moved range.
- • The ring update reaches routers at different times, so multiple rings are live simultaneously.
- • The data transfer is slower than the ring propagation, so the new owner is authoritative before it has the data.
- • The departing node leaves before its successor has finished pulling the range.
- • A node flaps in and out, causing arcs to move back and forth without ever settling.
- • Two implementations compute node positions differently and construct different rings from identical membership.
- • Phantom deletions during a join: for the duration of the transfer, a fraction of reads for existing keys return empty. The operator sees a spike in application-level "not found" handling — new default rows created, duplicate records written — with a zero error rate in every service.
- • Lost writes at the trailing edge: writes accepted by the old owner after its range was streamed are discarded at handover. The operator finds them missing during reconciliation, days later, with no log line at the moment of loss.
- • Successor overload on node removal: one survivor takes the whole departed node’s keyspace and doubles in load and disk usage. The operator sees a single node’s latency and utilisation diverge sharply while cluster averages barely move.
- • Stale-ring error curve: a decommissioned node’s address still receives traffic, and the connection-refused rate decays along the client fleet’s ring-refresh curve rather than dropping at the moment of removal.
- • Replication factor silently one: virtual node positions for the same physical machine sit adjacent on the ring, so all "three replicas" are the same box. The operator discovers it when that box dies and the data is gone.
- • Steady-state routing requires no coordination at all: given the ring, ownership is a pure function.
- • Agreeing on the ring requires either gossip (eventual, cheap, always briefly inconsistent) or consensus (ordered, expensive, correct). Which you need depends on whether ownership carries correctness weight or only routing weight — see Cluster Membership: A Belief, Not a Fact.
- • The handover itself needs a coordination point for the *moment* ownership flips, otherwise there is an interval with two owners. An epoch number committed through a single ordering authority is the cheapest way to get one.
- • Note the asymmetry that makes this affordable: coordination is required per *membership change* — a few times a week — not per request.
- • A network partition does not change any key’s position on the ring; only membership beliefs diverge. Both sides continue serving, each with its own idea of who owns the isolated node’s arc.
- • That means a partition on a gossip-based ring gives you two live owners for the isolated arc — writes accepted on both sides, with no ordering between them (Split-Brain: Two Nodes, Both Certain They Are In Charge).
- • If the ring is consensus-backed, the minority side cannot change membership and correctly refuses to reassign the arc, at the cost of that arc being unavailable to it.
- • Keys not in the affected arc are entirely unaffected in either design, which is the containment benefit of the whole scheme.
- • Detect: track ring epoch per node and per client. Any spread greater than one epoch during steady state means propagation is broken.
- • Contain: freeze ring changes during an incident. Automatic membership changes during overload are how a small problem becomes a rebalancing storm (Rebalancing: A Load Spike You Schedule for Yourself).
- • Recover: on a stalled transfer, keep the previous owner authoritative and restart the stream rather than promoting a partially-filled new owner.
- • Reconcile: after any change with a suspected write-loss window, run an Anti-Entropy: Repairing Divergence Nobody Reported pass over the affected arcs — this is exactly what that mechanism exists for.
- • Verify: assert that for every key range, exactly one node reports itself as owner and R distinct physical nodes report themselves as replicas.
- • Ring epoch distribution across the fleet — one number that makes the entire propagation window visible.
- • Requests answered with "not owner", by key range and by client ring epoch.
- • Bytes streamed per membership change, and which pair of nodes carried it. A single-pair transfer of a full node’s data is your warning that virtual nodes are missing or too few.
- • Per-node key-space share, computed from the ring rather than measured from disk — it tells you the imbalance before the data arrives.
- • Count of key ranges whose R replicas do not resolve to R distinct physical hosts. This should be zero and is not always.
- • Clusters that change size often — autoscaled caches, elastic storage tiers — where a full reshuffle per change is intolerable.
- • Caches specifically: a ring change invalidates only the moved arc rather than the entire cache, which is the difference between a small dip and a stampede against the origin (One Key Expires and Five Hundred Instances Miss at the Same Millisecond).
- • Leaderless replication systems where every node must independently compute the same replica set for a key without asking anyone (Leaderless Replication: Every Replica Accepts Writes).
- • Any design where you would rather have no partition map to keep consistent than have a small one.
- • Small clusters. With five nodes the arc imbalance is severe and virtual nodes are mandatory, at which point a fixed-partition map is simpler and more controllable.
- • When you need deliberate placement — pinning a tenant to specific hardware, or draining one node gradually. A ring gives you no handle for that; an explicit assignment map does.
- • When ownership carries correctness weight and membership is gossiped, because the window with two believed owners is then a window with two writers.
- • Range-scan workloads, since the ring destroys order exactly as a plain hash does.
- • A fixed number of partitions plus an explicit partition → node map. Same movement properties, plus deliberate placement and easier draining, at the cost of maintaining a small piece of agreed state — usually the better choice when you already have a coordination service.
- • Rendezvous (highest random weight) hashing: compute
hash(key, node)for every node and take the maximum. Same K/N movement, naturally better balanced without virtual nodes, and trivially supports weights — at O(N) per lookup instead of O(log N), which is fine for small N. - • Jump consistent hash: tiny, allocation-free, perfectly balanced, but nodes can only be added or removed at the end of the list — excellent for a numbered shard count, unusable for arbitrary membership.
- • Maglev hashing: builds a lookup table for near-perfect balance and minimal disruption, designed for load balancers where lookups vastly outnumber membership changes.
- • For a cache specifically: no ring at all. Let each node have its own local cache and accept duplication. Simpler, and duplication costs memory rather than correctness.
The ring: a key belongs to the next node clockwise
What people believe, and what is true
Consistent hashing means no data has to move.
It means the *minimum* moves — K/N rather than nearly K. A node joining still pulls a full share of data across the network.
Consistent hashing balances load across nodes.
It balances nothing on its own. Random positions produce very unequal arcs, and it says nothing at all about request rate per key.
Once the ring is updated, the change is done.
The ring update and the data transfer are separate events with a gap between them, and the ring update itself reaches different routers at different times. The gap is where the bugs are.
The successor absorbing a departed node is fine because it is only 1/N more data.
On a plain ring it is not 1/N more — it is 100% more, because one node inherits the whole arc. Only virtual nodes spread it.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Put keys and nodes on the same circle; a key belongs to the next node clockwise. Membership changes then disturb only one arc instead of the whole mapping.
Practical
Never let a node serve or 404 a key it may not own — respond "not owner, refresh". During a transfer, have the incoming owner forward unfetched keys to the outgoing one. Both rules cost a little latency and remove an entire class of silent data bugs.
Advanced
Version the ring with a monotonic epoch and carry it on every request. A node that sees an older epoch for a range it now owns rejects rather than serving; a node that sees a newer one refreshes. This turns the propagation window from an invisible correctness hazard into an explicit, observable retry.
Internals
Whether the two-owner window is a correctness problem or a routing inconvenience depends entirely on how membership is agreed. Under gossip, two partitions can each believe they own an arc indefinitely and both accept writes — you need version vectors or CRDTs downstream to survive it. Under consensus-backed membership with epoch fencing, the window is bounded by the commit and the old owner is provably unable to accept writes after it.
Apply it
- 🔧 Implement the ring lookup and verify empirically that adding an eleventh node to ten moves close to 1/11 of a large key sample.
- 🔧 Add an epoch to your implementation and write the test that proves an old-epoch write to a moved range is rejected rather than accepted and dropped.
- ⚡ A cache cluster autoscales from 8 to 12 nodes at peak. Origin load spikes 4× for two minutes. Explain what fraction of the cache was invalidated and whether the ring behaved correctly.
- ⚡ Two services build the ring from the same membership list but route the same key differently. Where would you look first?
- 💬 Explain why consistent hashing moves K/N keys and modulo hashing moves nearly all of them.
- 💬 A node is removed from a five-node ring with one position per node. What happens to the remaining four, and why is that worse than the average would suggest?
- 💬 During a ring change, a client reads a key and gets a 404, then reads it again a second later and gets a value. What happened, and how would you make that impossible?