The question this answers
How far apart can two clocks in my fleet be, and what breaks at that distance?
Skew between two synchronised nodes is *typically* bounded by a few milliseconds to tens of milliseconds; it is not guaranteed to be bounded at all. Any correctness argument of the form "the clocks are within X" is an assumption about the operating environment, not a property of the system, and it fails exactly when you most need it — during network trouble, host restarts and VM migrations.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node knows its own clock reading and, at best, the offset its NTP client last computed against a server. It does not know the skew against any *peer*, because it has never talked to that peer about time. When a node compares its clock to a timestamp in an incoming message, it is comparing two unrelated readings and has no way to attribute the difference between them to skew, to network delay, or to genuine elapsed time.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Skew, and why it is unmeasurable locally
Two nodes, both running NTP, both convinced they are correct. Node A's clock reads 40 ms ahead of node B's. This is skew. Neither node observes it, because observing it would require comparing the two readings *at the same instant*, and there is no shared instant to compare them at — the only channel between them is the network, and the network adds an unknown delay.
This is the recurring shape of the domain: the quantity that determines correctness is precisely the one that is not locally observable. A node receives a message stamped 12:00:03.100, reads its own clock as 12:00:02.900, and has three completely different explanations for the 200 ms discrepancy: the sender is 200 ms fast, the message took negative time (impossible, so partially), or the sender is fast by some amount and the message took the rest. It cannot separate them.
You can bound the *round trip* — that is measurable. What you cannot do is split a round trip into its two halves without assuming they are equal. Every synchronisation protocol makes that assumption, and every asymmetric path breaks it.
Where skew becomes a bug
Skew does not cause problems on its own. It causes problems wherever two machines must agree that some *moment* has passed. That pattern is more common than it looks, and it hides inside things people do not think of as timing code.
Notice the shape shared by all of these: a value is written on one machine and evaluated against now() on another. Whenever you see that shape, the correctness depends on skew, and skew is not bounded.
- Token and certificate validity. An issuer stamps
exp; a verifier compares it to its own clock. A verifier a few minutes fast rejects freshly-issued tokens; a verifier slow accepts expired ones. This is why validators traditionally allow a leeway window — a skew allowance in the protocol. - Lease and lock expiry. A holder believes its lease is live; the grantor believes it expired and hands it to someone else. Two writers now believe they hold exclusive access — see The Stale Lock Holder: A Paused Process Does Not Know It Was Paused and Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely for the only sound fix, which does not involve clocks.
- Cache and TTL evaluation. An entry written with an absolute expiry on a fast machine looks stale immediately to a slow one, and stampede behaviour follows: One Key Expires and Five Hundred Instances Miss at the Same Millisecond.
- Timestamp-ordered merges. The fastest clock in the fleet wins every conflict, permanently and invisibly: Last Write Wins Is Data Loss You Chose by Default.
- Scheduled and windowed work. Two workers disagree about which hour a record belongs to and it is counted twice or not at all — a real cause of billing discrepancies.
- Replay-protection windows. A nonce cache keyed by time rejects legitimate requests from a skewed client, or accepts a replay from one.
The two ways skew actually gets large
Steady-state skew in a healthy fleet is small, and this lulls people into treating it as bounded. The distribution is not the problem; the tail is. Skew becomes large through two distinct mechanisms, and both correlate with the incidents you are already having.
Loss of discipline. The NTP daemon stops, or its upstream becomes unreachable, and the local clock free-runs at its native drift rate — tens of ppm, so seconds per day and growing linearly. This is silent: the machine keeps serving, keeps stamping, and reports nothing wrong. It is also correlated with network incidents, meaning your clocks diverge exactly when the rest of your system is already under stress.
Discontinuity. A VM is live-migrated, resumed from a snapshot, or a container starts on a host whose clock was never disciplined. The clock is wrong from the first instruction. When the daemon eventually notices, the correction is large enough to be applied as a step, and the machine's time jumps — possibly backwards, past values it has already handed out.
The practical consequence: never write a correctness argument that depends on a skew bound you cannot enforce. If a design says "assume clocks are within 500 ms", ask what happens at 5 seconds, and then at 5 minutes. If the answer is "silent corruption", the design is wrong even though it works today.
// skew-dependent: two machines, two clocks, one comparison
if (Date.now() > lease.expiresAtUnixMs) { takeOver() }
// correctness requires |skew| < the margin you left. You did not leave one.
// skew-independent: the resource itself rejects the stale holder
const token = acquire() // monotonically increasing fence
write(resource, data, { fence: token }) // resource refuses any fence < highest seen
// no clock appears in the correctness argument at allDesigning so skew cannot hurt you
The durable fix is not tighter synchronisation. Tighter synchronisation makes the bug rarer, which makes it harder to find and no less severe. The fix is to remove the clock from the correctness argument, and there are only a few ways to do that.
First, replace absolute deadlines with durations measured locally: instead of "valid until 12:00:05", say "valid for 30 seconds from receipt", measured on the receiver's monotonic clock (Never Measure a Duration With the Wall Clock). Skew cannot affect a measurement that never leaves one machine.
Second, replace time-based mutual exclusion with a fence: a monotonically increasing token that the protected resource itself checks. A slow holder with a stale fence is rejected by the resource, no matter what any clock believes (Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely).
Third, where you genuinely cannot remove the clock, make the skew allowance explicit and asymmetric in the safe direction. Lease grantors should wait *longer* than the lease before reassigning; validators should be lenient about nbf and strict about exp, or vice versa, depending on which error is worse. An explicit leeway is a stated assumption you can review; an implicit one is a latent bug.
Key points
- Skew is the instantaneous difference between two clocks, and no node can measure it against a peer.
- Delay and skew are indistinguishable from a single received message — you cannot separate them without assuming path symmetry.
- Skew becomes large through loss of NTP discipline (linear free-run) or discontinuity (VM resume, migration, cold start), both correlated with incidents.
- The dangerous pattern is: value written on machine 1, compared against
now()on machine 2. Expiry, leases, TTLs, windows and LWW all have this shape. - The fix is to remove the clock from the correctness argument — local durations, fencing tokens, explicit leeway — not to synchronise harder.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Each node disciplines its own clock independently against a time source; no node exchanges clock state with its peers.
- • Residual offset persists after each correction, of a size the client estimates but cannot verify.
- • The pairwise difference between any two nodes is the difference of their residuals, which is not bounded by either one's estimate.
- • A message carrying a timestamp crosses that difference plus an unknown transit delay, arriving as a single unattributable number.
- • The receiver compares that number to its own clock and makes a decision whose correctness depends on a quantity it never measured.
- • The NTP daemon dies or loses its upstream and the clock free-runs.
- • A large correction is applied as a step, moving the clock backwards past timestamps already emitted.
- • A VM resumes from a snapshot with a clock minutes or hours stale.
- • Asymmetric network paths bias every offset estimate in the same direction across a whole rack.
- • A misconfigured or malicious upstream time source moves a whole fleet coherently — synchronised, and synchronised wrong.
- • Sudden 401 cliff on one host: after a clock step forward, that host rejects every token as expired. The operator sees the error rate concentrated on a single instance, with valid credentials and no deploy.
- • Two lock holders: node A believes its lease is live, node B believes it expired and takes over. The operator observes interleaved writes from two workers that "cannot" run concurrently, and no lock service error.
- • Windowed aggregates that do not add up: an hourly rollup double-counts records near the boundary because two workers disagree on which hour it is. Sums are close but never exactly reconcile.
- • A trace where a span starts before its parent: the child service's clock is behind. The operator sees a negative-duration span or a child rendered outside its parent in the waterfall.
- • Silent conflict loss concentrated on one replica: the fastest clock in the fleet wins every timestamp-ordered merge, so one datacentre's writes systematically survive and another's systematically disappear.
- • Detecting skew requires coordination that nobody does by default: an explicit exchange in which two nodes measure each other, which still cannot separate skew from asymmetric delay.
- • Bounding skew usefully requires either specialised hardware (local GPS/atomic sources) or a wait proportional to the bound — see Two Timestamps Are Not an Ordering.
- • Every clock-free alternative moves the coordination somewhere visible: a fence is checked by the resource, a lease is renewed against a grantor. That visibility is the point — you can monitor a coordination point, and you cannot monitor an assumption.
- • Under skew, timestamp comparisons keep returning answers; they simply return the wrong ones, without error.
- • Anything using durations measured on a single machine's monotonic clock is unaffected.
- • Anything using causality — Lamport Clocks: Consistent With Causality, Blind to Concurrency, Vector Clocks: Buying Concurrency Detection at O(N), Version Vectors: Making the Conflict Visible — is unaffected, which is the whole argument for those mechanisms.
- • Fenced resources remain safe: a stale holder is rejected by the resource regardless of what its clock believes.
- • Detect: per-host offset alerting, plus a synthetic check that compares timestamps of the same logical event as recorded by two services.
- • Contain: fail a host's health check when its offset exceeds your leeway, so it drains rather than serving skewed decisions.
- • Recover: discipline the clock by slewing where possible; where a step is unavoidable, restart the processes that cached derived deadlines.
- • Reconcile: for expiry-driven failures, re-issue; for merge-driven failures, recover from version history if you kept one — and if you did not, this is the argument for keeping one.
- • Verify: after recovery, re-run the synthetic cross-service timestamp comparison and confirm the distribution has re-centred.
- • Per-host clock offset as a distribution across the fleet, alerting on the maximum, never the mean.
- • A counter of clock steps applied, with direction and magnitude, treated as an event worth paging on for write-path hosts.
- • Rate of "timestamp in the future" rejections — one of the few direct, application-visible measurements of skew you can get for free.
- • Negative span durations and parent-after-child orderings in traces, aggregated by service pair: this measures skew *in the units that hurt you*.
- • Token validation failures broken down by host, which turns a fleet-wide mystery into a single-host clock problem in one glance.
- • Reasoning about skew is essential wherever a deadline or expiry crosses a machine boundary: auth, leases, TTLs, replay windows, billing periods.
- • It is the argument you need when someone proposes ordering by timestamp, and the only argument that reliably changes their mind is a concrete number for the tail.
- • Adding generous skew leeway everywhere weakens the guarantee you were buying — a five-minute token leeway is a five-minute window for a stolen token.
- • Obsessing over skew where nothing depends on cross-machine agreement (display timestamps, log labels) is wasted effort.
- • Measure durations locally on a monotonic clock and never send an absolute deadline across a boundary: Never Measure a Duration With the Wall Clock.
- • Use fencing tokens rather than time for mutual exclusion — correctness then depends on a counter the resource checks, not on any clock: Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely.
- • Use logical time for ordering: Lamport Clocks: Consistent With Causality, Blind to Concurrency or Vector Clocks: Buying Concurrency Detection at O(N).
- • Where you must use clocks, state the leeway explicitly and make the asymmetry deliberate: grantors wait longer than holders, so the safe error is the one that happens.
- • Buy a bound: run local stratum-1 sources and expose the uncertainty to applications, then wait it out where ordering matters.
The skew window, and what falls into it
// skew-dependent: two machines, two clocks, one comparison
if (Date.now() > lease.expiresAtUnixMs) { takeOver() }
// correctness requires |skew| < the margin you left. You did not leave one.
// skew-independent: the resource itself rejects the stale holder
const token = acquire() // monotonically increasing fence
write(resource, data, { fence: token }) // resource refuses any fence < highest seen
// no clock appears in the correctness argument at allWhat people believe, and what is true
If both machines run NTP against the same server, they agree.
They each converge toward that server with their own residual error and their own path asymmetry. Agreeing with a third party approximately does not make two parties agree.
A five-minute leeway makes token expiry safe.
It makes it *work*. It also extends the lifetime of every leaked token by five minutes, which is the security cost you just paid for the timing assumption.
Skew is bounded because our monitoring shows it under 10 ms.
Your monitoring shows it for hosts that are up, reporting, and running the agent. The hosts that matter are the ones where that is not true.
Skew only matters at high scale.
It takes exactly two machines. A single VM resumed from a snapshot against one healthy peer reproduces every failure in this lesson.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Two clocks differ, nobody can measure by how much, and anything that compares a timestamp from one machine against now() on another depends on that unmeasured difference.
Practical
Audit for the pattern "written on machine 1, compared to now() on machine 2": tokens, leases, TTLs, replay windows, hourly boundaries. Convert absolute deadlines into locally-measured durations, protect exclusivity with fences, and monitor per-host offset with alerting on the maximum.
Advanced
A single message gives you one equation in two unknowns (skew and one-way delay), so no exchange determines skew without an assumption of symmetry. Round-trip protocols bound |skew| by roughly half the round trip *under that assumption*; asymmetric routing breaks it, and the resulting bias is systematic rather than random, so averaging does not remove it.
Apply it
- 🔧 Grep for absolute expiry timestamps crossing a service boundary. For each, write down what happens at 5 seconds, 5 minutes and 5 hours of skew.
- 🔧 Instrument one service pair with a synthetic event stamped by both sides and chart the difference. That chart is your real skew distribution.
- ⚡ After a hypervisor maintenance window, one instance in an autoscaling group starts failing every authenticated request while its siblings are fine. Health checks pass. What do you check first?
- ⚡ An hourly billing rollup is consistently off by a handful of records at the boundary and nobody can reproduce it locally.
- 💬 Your token validator starts rejecting freshly-issued tokens on one host. Walk through the diagnosis.
- 💬 A node receives a message whose timestamp is 200 ms in its own future. List every explanation, and say which the node can rule out.
- 💬 Why is "add a five-minute clock leeway" a security decision as well as a correctness one?