Coordination

Coordination Services: The Primitives, Not the Product

etcd, ZooKeeper and Consul are marketed as different products and are, underneath, the same four primitives over a consensus-backed key-value store. Learn the primitives — compare-and-swap, ephemeral keys, watches, leases — and every one of them becomes a configuration detail.

▶ Run the lab

The question this answers

The question

What does a coordination service actually give me that a database does not?

The guarantee — the property claimed, and its scope

A linearizable key-value store with atomic conditional writes, backed by consensus over a majority — so every client sees the same sequence of state changes, and no two clients can both succeed at a mutually exclusive operation. Available only while a majority of its members is reachable, and sized for small, frequently-read, rarely-written metadata rather than for application data.

Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.

What a node knows — observation versus inference

A client knows what the service told it at the moment of the reply. A watch tells it that something changed, not what the state is *now* — by the time the notification is processed, further changes may have occurred, so a watch must always be followed by a read. And a client cannot know its session is still alive: session expiry is decided by the service, and the client learns of it only when it next succeeds in communicating, which is exactly when it may be too late.

A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.

What guarantee?→What does a node know?→How does it work?→What can fail?→How does it fail?→Where is coordination?→What holds under failure?→How does it recover?→How would you know?→What is the simpler thing?
etcdzookeeperconsulprimitivescompare-and-swap

Four primitives, and everything else is built from them

Strip away the APIs and the product names and you find the same small set. Each primitive exists because it cannot be built safely from the others, and each maps directly onto a coordination problem from earlier in this module.

Everything a coordination service is used for — leader election, locks, membership, configuration, service registration — is a composition of these four. Once you see that, "which coordination service?" becomes a question about operations, ecosystem and familiarity rather than about capability.

  • Atomic compare-and-swap on a key. "Set this key to X only if its current version is V" (or only if it does not exist). This is the primitive that makes mutual exclusion possible; without atomicity, two clients can both read "unheld" and both write. It is the distributed analogue of compare-and-swap on a machine word, which Concurrency covers at the hardware level.
  • Ephemeral / leased keys. A key that disappears when its owner’s session or lease ends. This is how authority is revoked from a node you cannot reach, and it is Leases: Authority With an Expiry Date provided as a service rather than implemented by you.
  • Watches. A notification when a key or prefix changes, so clients learn of changes without polling. Notifications are hints, not state — always re-read.
  • Ordered, monotonic revisions. Every change gets a strictly increasing revision number, cluster-wide. This is your fencing token, already available, already ordered by consensus. See Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely.

Composing them: leader election in four lines

The classic recipes are short once the primitives are clear, and worth reading because they show exactly where each primitive earns its place.

The revision returned by the successful create is the fencing token. This is the part most implementations discard, and discarding it is what turns a correctness-grade election into an efficiency-grade one.

1# LEADER ELECTION
2lease = svc.grant_lease(ttl=15s)                  # ephemeral: dies with the session
3ok, rev = svc.put_if_absent("/service/leader", me, lease)
4if ok:
5    leader = true
6    token  = rev                                  # <-- the fencing token. Keep it.
7    svc.keep_alive(lease)                         # renew in the background
8else:
9    svc.watch("/service/leader")                  # learn when it is released
10    # on notification: re-read, then retry. The watch is a hint, not state.
11 
12# LOCK (same shape, different key)
13ok, rev = svc.put_if_absent("/locks/shard-7", me, lease)
14 
15# MEMBERSHIP: each member writes an ephemeral key under a prefix
16svc.put("/members/" + me, addr, lease)
17members = svc.list_prefix("/members/")            # dead members vanish on lease expiry
18 
19# CONFIGURATION: one key, many watchers, monotonic revisions
20cfg, rev = svc.get("/config/app")
21svc.watch("/config/app", from_revision=rev)       # no missed updates in the gap
Leader election and a lock, built from the primitives

What these services are not for

Every coordination service has the same failure story in production, and it is always the same shape: it was used as a database. The properties that make it trustworthy — every write goes through consensus, every write hits every member’s disk, the whole dataset is usually held in memory — are exactly the properties that make it a poor general-purpose store.

Concretely: keep the total dataset small (megabytes, not gigabytes), keep the write rate low (hundreds per second, not thousands), keep values small, and keep watch counts bounded. Exceeding these does not produce a gradual slowdown — it produces leader elections, which take out leadership and membership for everything else that depends on the service.

The consequence is a blast radius that is much larger than the abusing component. When etcd degrades, the Kubernetes control plane degrades; when ZooKeeper degrades, everything that elects a leader through it degrades. A coordination service is a shared dependency of last resort, and treating it as application storage couples every one of its consumers to your worst-behaved one.

PropertyCoordination serviceRelational database
Dataset sizetypicalMegabytesTerabytes
Write rateprotocolLow — every write is a consensus roundHigh
ConsistencytypicalLinearizable by defaultConfigurable; often read-replica stale
Change notificationtypicalNative watchesPolling or CDC
Automatic key expirytypicalNative — ephemeral keysApplication-managed
On losing a majorityprotocolRead-only or unavailableDepends on replication setup
Coordination service versus database — different tools

The operational realities that bite

Three things surprise teams operating one of these for the first time, and none of them are visible from the API.

It is a hard dependency for everything built on it. If it is unavailable, leadership cannot change, locks cannot be acquired, and members cannot register. Design consumers to keep working on their last-known state where possible — a service that already holds a valid lease should continue until it expires, rather than failing immediately.

Watches are not a state channel. A watch may coalesce events, and a client that reconnects can miss changes unless it resumes from a revision. Always: watch, then read; and where the API supports resuming from a revision, use it. Treating watch payloads as authoritative state is the most common client bug.

Sessions expire in ways clients under-handle. When a session expires, every ephemeral key it owned vanishes — leadership, locks, registration, all at once. A client that treats "session expired" as a reconnect rather than as a total loss of authority will keep acting as leader after its leadership key is gone. That is The Stale Lock Holder: A Paused Process Does Not Know It Was Paused with the service’s name on it.

Key points

  • Coordination services are consensus-backed key-value stores exposing four primitives: atomic compare-and-swap, ephemeral/leased keys, watches, and monotonic revisions.
  • Leader election, locks, membership and configuration are all compositions of those four.
  • The revision returned on a successful write is your fencing token — keep it.
  • A watch is a hint that something changed; always re-read, and resume from a revision after reconnect.
  • They are sized for small, low-write metadata; using one as a database degrades every consumer.
  • Session expiry removes every ephemeral key at once and must be handled as total loss of authority.

The chain, answered

Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.

How it works
  • • The service replicates a key-value store across an odd number of members using consensus (Raft in etcd and Consul; Zab in ZooKeeper).
  • • Writes are proposed to the leader, committed once a majority has them, and assigned a strictly increasing revision.
  • • Reads are served linearizably — via the leader, or from a follower with a read-index or lease confirmation.
  • • Clients hold a session or lease, renewed by keep-alives; ephemeral keys are tied to it.
  • • When a session lapses, all its ephemeral keys are deleted, and watchers on those keys are notified.
  • • Clients use conditional writes to claim exclusive keys and use the returned revision as a fencing token.
What can fail at the boundary
  • • The service loses its majority and becomes unavailable for writes, so no leadership or lock can change hands.
  • • A client’s session expires during a network blip, silently removing its leadership and locks.
  • • Watch notifications are missed across a reconnect, leaving a client acting on stale state.
  • • The dataset or write rate grows past what the service is sized for, causing elections inside the service itself.
  • • A client treats a watch payload as current state and acts on a value that has already changed.
  • • Clients retry aggressively during a service outage, adding load to a service that is already struggling.
How it fails — what an operator sees
  • • Control-plane freeze: the service loses quorum and every dependent system stops changing state while continuing to serve existing traffic. The operator sees deploys hanging, leadership stuck, and application traffic unaffected — a confusing signature.
  • • Silent leadership loss: a client’s session expires, its leadership key is deleted, another node takes over, and the original keeps acting. The operator sees two active leaders and a client whose logs show only a reconnect.
  • • Service degraded by a single abusive consumer: one team stores large values or writes at high rate; leader elections begin inside the coordination service; every unrelated consumer loses leadership simultaneously. The operator sees a fleet-wide incident traceable to one key prefix.
  • • Watch storm: thousands of clients watch the same key and all wake on every change, then all read. The operator sees load spikes on the service synchronised with configuration changes.
  • • Stale action after reconnect: a client resumes a watch without a revision, misses an update, and acts on old configuration. The operator sees a subset of nodes behaving according to a previous config with no error anywhere.
Where coordination is required
  • • Every write is a consensus round — this is why write throughput is low and why it should stay that way.
  • • Reads can be cheap if the service uses leader leases; linearizable reads still cost a confirmation.
  • • Consumers should amortise: hold a lease and act locally, rather than consulting the service per operation.
What still holds under failure
  • • On losing a majority the service refuses writes and may serve stale reads, depending on configuration; it does not diverge.
  • • Existing leases continue to be honoured by their holders until expiry, so a short outage need not stop work if consumers are designed for it.
  • • Anything requiring a *change* of authority stops until the majority returns.
How it recovers
  • • Detect: alert on the service’s own quorum health and leader-change rate, separately from client-side errors.
  • • Contain: isolate consumers so one cannot exhaust the service; enforce quotas on key count, value size and write rate per prefix.
  • • Recover: restore majority membership; clients re-establish sessions and re-acquire keys automatically.
  • • Reconcile: after a session expiry, clients must verify they still hold what they think they hold before acting — never assume across a reconnect.
  • • Verify: game-day the service’s unavailability and confirm consumers degrade as designed rather than failing immediately.
How you would know
  • • Quorum health and leader-change rate of the coordination service itself.
  • • Database size, key count, and write rate against the service’s documented limits.
  • • Session expiry events per client, and what each client did afterwards.
  • • Watch count and notification fan-out per key prefix.
  • • p99 write latency, which rises sharply well before the service fails outright.
When it helps
  • • Leader election, membership, distributed locks and configuration where a real guarantee is needed and you do not want to implement consensus.
  • • Anywhere you need change notification with ordering, which a database gives you only awkwardly.
  • • Anywhere authority must be revoked from an unreachable node, via ephemeral keys.
When it hurts
  • • As application storage — the fastest way to destabilise every consumer at once.
  • • For high-frequency coordination, where the per-write consensus cost dominates.
  • • When a database you already run could enforce the same constraint with a unique index or a transaction.
  • • When it becomes an undeclared dependency of every service, so its availability caps the whole platform’s.
Simpler alternatives
  • • A relational database with a unique constraint or advisory lock, when the state already lives there and you already operate it well.
  • • Your platform’s built-in primitive — Kubernetes Lease objects, a cloud provider’s lock service — which is the same thing without a new system to run.
  • • Consensus embedded in your own service (a Raft library), avoiding an external dependency at the cost of operating consensus yourself. Rarely the right trade.
  • • No coordination service at all, via partitioned ownership. See Coordination Avoidance: Restructuring the Problem Instead of Paying for It.

The four coordination primitives

The primitives, not the product
etcd, ZooKeeper and Consul are marketed as different products and are, underneath, the same four primitives over a consensus-backed key-value store. Learn the primitives and each product becomes a configuration detail.
primitive
what it gives you
An atomic conditional write: set this key to V only if its current version is exactly X. One of two racing clients succeeds and the other is told it lost.
what you build from it
Locks, leader election, uniqueness, optimistic updates — everything in this list reduces to it.
the trap
Discarding the revision the write returns. That revision is your fencing token, and without it you have coordination with no protection at the resource.
ok, rev = kv.put("lock/shard7", me, if_version=0, ttl=15s)
if not ok:
    # someone else holds it; do not proceed
    return
build
✓ compare-and-swap· ephemeral key / session· watch✓ lease + revision
CAS on an empty key with a lease attached gives at-most-one-holder-of-record and automatic release on crash. The returned revision is the fencing token — and without checking it at the resource this is an efficiency lock, not a correctness one.
Every write here is a consensus round replicated to every member’s disk. That is why write throughput is low and why it must stay that way: one team storing large values or writing at high rate starts leader elections *inside* the coordination service, and every unrelated consumer loses leadership simultaneously. Keep the dataset small, enforce per-prefix quotas, and amortise — hold a lease and act locally rather than consulting the service per operation.
Using a coordination service does not make your locks safe. It gives you a correct lock-of-record and a revision you can fence with. Discard the revision, or fail to compare it at the resource, and you have exactly the stale-holder exposure of any other lock — see the fencing-token lab.
protocolLinearizability and atomic conditional writes follow from the consensus layer beneath. Both hold only while a majority of the service’s own members is reachable; below that it is unavailable for writes, by design.
assumptionClient-side correctness assumes sessions are handled as *authority*, not as connections. A client that treats session expiry as a reconnect keeps acting on revoked authority.
typicalPractical limits — a few gigabytes of data, low-thousands of writes per second — vary by service and version. The failure past them is leader instability inside the coordination service, which takes down every consumer at once, not a clean rejection.
simplifiedThe snippets omit retry, backoff, session re-establishment and revision bookkeeping, all of which a production client library handles and a hand-rolled client usually does not.

What people believe, and what is true

Claim

etcd, ZooKeeper and Consul are fundamentally different tools.

Reality

They expose the same four primitives over consensus. Choose on operations and ecosystem, not on capability.

Claim

It is a fast key-value store, so we can put application state in it.

Reality

Every write is a consensus round replicated to every member’s disk. Volume causes leader instability, which takes down every consumer at once.

Claim

A watch tells me the current value.

Reality

It tells you something changed. Always re-read, and resume watches from a revision so reconnects do not silently skip updates.

Claim

Using a coordination service means my locks are safe.

Reality

It gives you correct locks-of-record and a revision you can fence with. If you discard the revision and do not check it at the resource, you have the same stale-holder exposure as any other lock.

Go deeper

Only the levels this lesson can honestly fill — a missing level is a claim nobody had.

Overview

A coordination service is a small, consensus-backed key-value store with atomic conditional writes, keys that vanish when their owner dies, change notifications, and ordered revisions. Everything else is built from those four.

Practical

Use conditional writes for exclusivity, ephemeral keys for revocable authority, and the returned revision as a fencing token. Always re-read after a watch and resume from a revision. Keep the dataset small and the write rate low, enforce per-prefix quotas, and treat session expiry as total loss of authority.

Advanced

The service is a replicated state machine exposing a linearizable KV interface, so its guarantees and its limits are exactly those of The Raft Log: Commit Index, Divergence and Reconciliation: majority required, writes serialised through one leader, reads linearizable only via the leader or a read-index. Every client-side coordination pattern you build is a reduction to the single decision the store can make atomically — a conditional write — which is why compare-and-swap on a key is the primitive everything else composes from.

Apply it

Build it, then break it
  • 🔧 Build lock, leader election and membership from the four primitives, and identify which primitive each one depends on most.
  • 🔧 Audit a coordination-service client in your system for watch handling: does it re-read, and does it resume from a revision?
Interview questions
  • 💬 What primitives does a coordination service provide, and what can you build from each?
  • 💬 Implement leader election using them, and say where the fencing token comes from.
  • 💬 Why should you not store application data in etcd or ZooKeeper?
  • 💬 A client reconnects after a network blip. What must it check before continuing to act as leader?
← The Stale Lock Holder: A Paused Process Does Not Know It Was PausedDistributed Uniqueness: One Name, Many Shards →