Conflict Resolution

Version Vectors: Making the Conflict Visible

A vector clock scoped to one object, tracked at the replicas rather than at every client. It answers one question exactly — does this version supersede that one, or are they concurrent — and answering it is what turns silent data loss into an explicit decision.

▶ Run the lab

The question this answers

The question

Given two versions of an object, how do I know whether one supersedes the other or they conflict?

The guarantee — the property claimed, and its scope

For a single object, a version vector determines the causal relationship between two versions exactly: one dominates (a supersession), the other dominates, they are equal, or neither dominates (a genuine conflict). This holds as long as every writing replica has its own component and no component is removed. It says nothing about which version is *better*, and nothing about objects other than the one it stamps.

Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.

What a node knows — observation versus inference

A replica knows, per object, how many writes from each replica it has incorporated. VV[r] = 4 means "this version includes the first four writes replica r made to this object" — knowledge about the version's ancestry, not about replica r's current state. When a client presents a version vector with a write, the replica knows exactly which state that client had read, and therefore whether the write is an update or a concurrent branch.

A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.

What guarantee?What does a node know?How does it work?What can fail?How does it fail?Where is coordination?What holds under failure?How does it recover?How would you know?What is the simpler thing?
version vectorscausalityconflict detectionsiblingsdotted version vector

Same idea as a vector clock, three differences that matter

A version vector is a Vector Clocks: Buying Concurrency Detection at O(N) specialised for replicated data. The algorithm is the same — a per-replica counter, element-wise max on merge — and the differences are all about scope and cost.

Per object, not per node. The vector stamps a value, so the counters advance only when *that object* is written. An object written twice carries a tiny vector regardless of how busy the cluster is.

Indexed by replica, not by actor. The components correspond to storage replicas, of which there are three or five, rather than to clients, of which there may be millions. This is the change that makes the mechanism affordable, and it is why version vectors ship in real systems where general vector clocks do not.

Carried by the client through a read-modify-write. The client reads a value with its vector (often opaquely, as a "context"), modifies it, and sends the vector back with the write. That returned vector is the evidence of what the client saw — which is exactly what lets the replica distinguish "this client is updating the version it read" from "this client never saw the version I have".

1// client
2(value, ctx) = get(key) // ctx carries the version vector
3new = modify(value)
4put(key, new, ctx) // hand the evidence back
5
6// replica r, on put(key, new, ctx)
7switch compare(ctx, stored.vv):
8 case ctx >= stored.vv: // client saw everything we have
9 stored.value = new
10 stored.vv = increment(ctx, r) // clean update, no conflict
11 case ctx < stored.vv: // client saw an older state
12 // stale write: either reject (optimistic concurrency)
13 // or keep both (sibling) — a product decision, not a technical one
14 case CONCURRENT: // neither dominates
15 siblings.add(new, increment(ctx, r)) // <- the conflict, made visible
16
17// a later read returns the sibling set; resolving and writing back
18// with the combined context collapses it to one version again.
The read-modify-write cycle, and the three-way outcome

Siblings: what "keeping both" actually looks like

When a conflict is detected, the store does not choose. It keeps both versions as siblings under the same key, and the next read returns a set rather than a value. This is Dynamo's design and it remains the clearest expression of the idea: the store's job is to preserve, the application's job is to decide.

The consequences are worth being concrete about, because "just keep both" hides real work. Every read path must handle a set of size ≥ 1. Every client must be able to merge, or at least present a choice. And siblings only disappear when somebody resolves them and writes back with the combined context — a read that ignores siblings leaves them in place forever.

That last point produces the characteristic failure: sibling explosion. If nothing resolves, each new conflicting write adds another sibling, objects grow, reads slow down, and eventually a size limit is hit. The pathological version is a client that reads siblings, ignores them, and writes a fresh value *without* the combined context — which conflicts with all of them and adds one more. Resolution must be a real, exercised path, not a TODO.

There is one sibling that behaves unlike the others and catches people: a delete. If one replica deletes an object and another concurrently writes it, the merge sees a conflict between "gone" and a value. Deleting the record entirely loses the fact that a delete happened, so the write resurrects the object at the next anti-entropy pass. The fix is a tombstone — a deletion recorded as a version, with its own place in the vector — which then has to be garbage-collected safely, and that is a coordination problem of its own (Anti-Entropy: Repairing Divergence Nobody Reported).

Two clients read the same version, write concurrently, and produce siblingsprotocol
Client 1Replica setClient 2put(+lamp, ctx=[1,0]): deliveredput(+lamp, ctx=[1,0])put(+mug, ctx=[1,0]): deliveredput(+mug, ctx=[1,0])cart={book} vv=[1,0] (write) at t=0cart={book} vv=[1,0]read: ctx=[1,0] (read) at t=2read: ctx=[1,0]read: ctx=[1,0] (read) at t=3read: ctx=[1,0]put +lamp ctx=[1,0] → vv=[2,0] (write) at t=6put +lamp ctx=[1,0] → vv=[2,0]put +mug ctx=[1,0] → CONCURRENT → sibling (decide) at t=8put +mug ctx=[1,0] → CONCURRENT → siblingt=0time →t=8
delivereddelayed (dashed, long)duplicated (×2)dropped — stops short, never arriveswritereaddecide
Client 2's context proves it never saw the lamp write, so its write is not an update — it is a branch. Without the context the replica would see two puts arriving in order and would simply overwrite. The evidence the client carries is the entire mechanism.

The sibling-per-write problem, and dotted version vectors

A plain version vector has a defect that shows up under load, and the fix is the reason modern implementations look more complicated than the textbook version.

The issue: a version vector summarises a *set* of writes, but a stored value is a *single* write. When two clients write concurrently against the same context, the replica must represent "these two specific writes are concurrent" — and a vector of maxima cannot distinguish "I have replica A's writes 1 and 2" from "I have replica A's write 2 only". The practical consequence is false conflicts: sequential writes from the same client can be reported as concurrent, and each one spawns a sibling. Under a workload of repeated updates to one key, siblings accumulate for no semantic reason at all.

Dotted version vectors fix this by attaching, to each stored version, both the vector (what it has seen) and a dot — the single (replica, counter) pair identifying the write that produced it. Comparison then asks whether one version's dot is contained in the other's vector, which distinguishes "this write is included in what you have" from "this write is a branch". The result is that the sibling count reflects genuine concurrency rather than an artefact of the encoding.

The practical advice is short: if you are implementing this yourself, implement dotted version vectors, and test the specific case of a client doing repeated sequential writes to one key. If a plain version vector produces siblings there, your users will see them constantly.

Metadata sizeFalse conflicts?Use when
None (LWW)protocol0N/A — no detection at allNever, for data you care about
Single version numberprotocolO(1)Cannot detect concurrency; only stalenessSingle-writer optimistic concurrency
Version vectortypicalO(replicas)Yes — sequential writes can look concurrentSmall replica sets, low update rate per key
Dotted version vectortypicalO(replicas) + one dotNoThe default for a real leaderless store
Full vector clock (per client)typicalO(clients)NoOnly when per-client causality is genuinely required
Choosing a detection mechanism

What detection buys you, and what it does not

The value of version vectors is not that they solve conflicts. It is that they convert an invisible, unrecoverable loss into a visible, deferrable decision — and the difference between those two situations is enormous operationally.

With detection you get: a conflict *count* you can put on a dashboard and argue about; both values preserved, so any later decision is still possible; and an explicit place in the code where the resolution rule lives, which can be reviewed and tested. Without it you get a number that is always zero and a support queue.

What you do not get is the rule itself. A version vector will tell you that "Q3 Plan" and "Q3 Planning" are concurrent; it has no opinion about which a user wants, and cannot have one. That is Only the Application Knows What the Merge Means. Nor does it help with causality that travelled outside the system, and nor does it extend across objects — two objects with an invariant between them are not protected by per-object vectors, which is why multi-object invariants need coordination rather than better metadata (Start From the Invariant, Not From the Architecture).

Key points

  • A version vector is a vector clock scoped to one object and indexed by replica, which is what makes the metadata affordable.
  • The client carries the vector through a read-modify-write; that returned context is the evidence of what the client actually saw.
  • Comparison yields supersession, staleness, equality, or genuine concurrency — the last one is the conflict.
  • Detected conflicts are kept as siblings. Siblings only disappear when something resolves them and writes back with the combined context.
  • Concurrent delete and write require tombstones, or the delete is undone by the next anti-entropy pass.
  • Plain version vectors produce false conflicts on sequential writes; dotted version vectors fix this and are what real systems use.
  • Detection makes the loss visible and deferrable. It does not supply the resolution rule.

The chain, answered

Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.

How it works
  • Each object carries a vector with one counter per replica that has ever written it.
  • A read returns the value (or sibling set) together with the vector as an opaque context.
  • A write carries that context back, so the replica learns exactly which state the client observed.
  • The replica compares the incoming context to the stored vector: dominance means update, dominated means stale, mutual non-dominance means conflict.
  • On conflict the new version is stored as a sibling; on resolution the merged value is written with a context covering all siblings, collapsing them.
What can fail at the boundary
  • The client drops the context (a naive client library, a proxy that strips it), so every write looks concurrent and siblings multiply.
  • A replica identifier is reused after a restart, merging two distinct histories into one component.
  • Tombstones are garbage-collected too early and a deleted object is resurrected by a lagging replica.
  • The vector grows as replicas are added and removed, and unsafe pruning turns real conflicts into apparent supersessions.
  • Siblings are never resolved by any read path, and objects grow until a size limit rejects writes.
How it fails — what an operator sees
  • Reads start returning many versions: p99 response size for a key range climbs while request rate is flat. The operator finds a client that reads siblings and ignores them.
  • Deleted records come back: an object is deleted, disappears, and reappears minutes later. The operator sees a delete with a success response and an object that exists — the tombstone was dropped or never created.
  • Writes rejected at the size limit on the busiest keys: accumulated siblings and metadata push objects past a maximum, so the hottest, most contended keys are the first to become unwritable.
  • Conflict count collapses to zero after a client-library upgrade: the new library stops returning the context, so every write is treated as an unconditional overwrite. The metric improving is the bug.
  • Constant false conflicts from one client: a client doing repeated sequential updates to one key generates a sibling per write. The operator sees sibling growth correlated with a single caller and no genuine concurrency.
Where coordination is required
What still holds under failure
  • Both sides of a partition continue accepting writes and stamping them correctly; nothing is lost.
  • After healing, every concurrent pair is correctly identified as a conflict rather than silently resolved.
  • A crashed replica's component freezes, remaining correct for comparison and becoming permanent metadata overhead.
  • A version separated from its vector is unusable — the metadata must be stored and replicated atomically with the value.
How it recovers
  • Detect: track sibling count per read and conflict rate per key range; both are quiet until they are not.
  • Contain: enforce a sibling cap with an explicit policy at the limit, so a runaway key degrades in a chosen way rather than by rejecting writes.
  • Recover: run a resolution pass over keys with siblings, applying the application merge rule and writing back the combined context.
  • Reconcile: for objects resurrected by tombstone loss, re-apply the delete and fix the GC watermark that allowed it.
  • Verify: after resolution, confirm sibling counts return to one and that replica digests agree for the affected range.
How you would know
  • Siblings per read, at max and p99 — the direct measure of unresolved conflicts.
  • Conflict detections per second by key range and by client, which identifies the caller generating them.
  • Object size distribution including metadata, so you see the size-limit wall before you hit it.
  • Tombstone count and age, plus the GC watermark, to catch premature collection before a resurrection happens.
  • Rate of writes arriving with no context, which is the signal that a client is bypassing the mechanism entirely.
When it helps
When it hurts
  • Single-leader systems, where the leader already orders writes and a simple version number is sufficient.
  • Very small values written very frequently, where per-object metadata rivals the payload.
  • Applications with no resolution path — you have paid for detection and gained a growing sibling set.
Simpler alternatives

Does this version supersede that one, or do they conflict?

Does this version supersede that one, or do they conflict?
Two versions, one object. A version vector answers the relationship exactly — dominates, is dominated, is identical, or is concurrent — and the last of those is the only one that is a conflict.
r1 versions held
1
r2 versions held
1
relationship
equal
conflict detected
no
r1
title = "Untitled"{}
r2
title = "Untitled"{}
Both replicas hold one version with an empty vector — nobody has written yet.
No conflict at this point
Every version here is an ancestor or a descendant of the others, so the relationship is a supersession and the store can resolve it with no help from you. Write on both replicas without syncing to produce the case that actually needs a decision.
A version vector is a vector clock scoped to one object and indexed by replica — which is what makes the metadata affordable, since the number of replicas is small and the number of clients is not. VV[r] = 4 means "this version includes the first four writes replica r made to this object": a statement about the version's ancestry, not about r's current state. The client carrying the returned context through a read-modify-write is the load-bearing part, because that context is the evidence of what the client actually saw. Plain version vectors also produce false conflicts on ordinary sequential writes, which is why real systems use dotted version vectors. And note the limit: detection makes the loss visible and deferrable. It does not supply the resolution rule.
protocolDominance is the definition, not a model: it is decidable from the vectors alone, and it holds as long as every writing replica has its own component and no component is ever removed. What it never tells you is which version is better.

What people believe, and what is true

Claim

Version vectors resolve conflicts.

Reality

They detect them exactly. Resolution needs a rule the vector cannot supply, and picking a sibling by timestamp reintroduces LWW.

Claim

Siblings are an error condition.

Reality

They are the store correctly preserving both writes. The error is having no path that resolves them.

Claim

Deleting the record handles a delete.

Reality

Without a tombstone the delete has no version, so a concurrent write resurrects the object at the next repair.

Claim

A version number is basically the same thing.

Reality

A scalar detects staleness but cannot represent "neither is newer". Concurrency needs a partially ordered structure.

Claim

Our conflict rate is zero, so we have no conflicts.

Reality

Check whether anything can detect one. A zero that has never been non-zero is usually a measurement gap.

Go deeper

Only the levels this lesson can honestly fill — a missing level is a claim nobody had.

Overview

One counter per replica, per object. The client carries it through a read-modify-write, and the store uses it to tell an update from a branch. Branches are kept as siblings rather than discarded.

Practical

Make sure your client library returns the context, that some read path resolves siblings and writes back, and that deletes create tombstones. Monitor siblings-per-read and conflict rate; a conflict rate that is always zero usually means detection is not happening.

Advanced

The plain vector summarises a set of writes while a stored value is one write, which is why sequential writes can appear concurrent. Dotted version vectors attach the originating (replica, counter) dot to each version, so containment of the dot in the peer's vector distinguishes inclusion from branching. This removes false siblings and is the standard implementation — and its edge cases around replica removal and tombstone GC are where implementations most often go subtly wrong.

Apply it

Build it, then break it
  • 🔧 Implement the three-way comparison and a resolution path that writes back a combined context; verify siblings collapse to one.
  • 🔧 Write a test where a single client performs ten sequential updates to one key, and assert no siblings are created. A plain version vector will fail it.
Reason about this
  • After upgrading a client library, your conflict metric drops to zero and stays there. Is this good news?
  • A deleted user record reappears in search results twenty minutes after deletion, twice a month. Explain the mechanism.
Interview questions
  • 💬 How does a version vector distinguish a stale write from a concurrent one?
  • 💬 Two replicas: one deletes an object, the other writes it. What must the system store to get this right?
  • 💬 Your sibling count is climbing steadily. List the possible causes in order of likelihood.