The question this answers
A value changed at the source. Which copies are stale, and what actually tells them?
With TTL alone: staleness is bounded by the TTL, unconditionally and without any messaging. With invalidation added: *expected* staleness drops to the delivery latency, but the *maximum* is still the TTL — and only if the fill path is ordered against invalidation. Without that ordering, staleness is unbounded.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A cache node knows what it holds and when it stored it. It cannot know whether the source has changed since, because nothing about a stored value carries information about later writes. It learns of a change only if a message reaches it — so an invalidation that is dropped, or that arrives before the write it invalidates has been read, leaves a node confidently serving a value it has no way to suspect.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The race that produces permanent staleness
This is the bug worth memorising, because it is the reason experienced engineers are suspicious of caches. It requires no lost messages and no clock skew — only ordinary interleaving.
A reader misses on key k and queries the database, which returns the old value v1. Before that reader gets around to writing to the cache, a writer updates the database to v2 and sends an invalidation for k. The invalidation arrives at a cache that does not currently hold k, so it does nothing. *Then* the slow reader completes and writes v1 into the cache with a fresh TTL.
The cache now holds the old value, the invalidation has already been consumed, and there is nothing left to correct it. The cache serves `v1` until the TTL expires — which may be hours — and every observable signal is healthy: no errors, no lag, delivery worked perfectly, and the invalidation was processed exactly as designed.
Notice what this says about the general problem. Invalidation is a message ordered against a *fill* that is itself the result of an earlier read. Those two operations race, and the ordering between them is not something the messaging layer can establish, because the fill did not exist when the invalidation was sent. This is why "just publish invalidations" is not a solution and why the mitigations below are all about making the fill path safe rather than making delivery better.
What each delivery semantics buys you
Invalidation inherits the delivery semantics of whatever carries it, and each choice fails differently. At-most-once (fire-and-forget, best-effort pub/sub) means a dropped invalidation leaves a node stale until TTL, with no error and no way to detect it. At-least-once (durable topic with acknowledgements) means duplicates, which for a delete are harmless — invalidation is naturally idempotent, one of the few genuinely pleasant facts in this area. Ordering matters only per key, and per-key ordering is usually obtainable by partitioning on the key.
Then there is the fan-out question. With per-instance caches, an invalidation must reach *every* instance, so it is a broadcast: 500 instances mean 500 deliveries per invalidation, and a bulk update producing 100,000 invalidations means 50 million deliveries. The messaging system becomes a bottleneck long before the cache does, which is You Cannot Enumerate the Caches, So TTL Is the Bound and Invalidation Is the Optimisation.
A cleaner alternative sidesteps delivery entirely. Version the key: instead of invalidating product:42, write to product:42:v7 and bump the version. Readers construct the key from a version they read alongside the data, so a stale reader simply reads an old key that nobody will write to again, and eviction reclaims it. There is no invalidation message, therefore no delivery semantics and no race — you have converted a coherence problem into a naming problem. The cost is a version lookup on the read path, and garbage.
The third family is write-through with compare-and-set: the writer sets the new value rather than deleting, and the set is conditional on the version already present. The slow reader’s SET v1 fails because the cache holds a newer version, which closes the race directly. This requires versioned values and a cache that supports conditional writes, and it is the most robust option where both are available.
| Strategy | Max staleness | Messaging needed | Characteristic failure |
|---|---|---|---|
| TTL onlyprotocol | TTL | None | Everyone is stale up to the TTL, always, by design |
| Delete on write (at-most-once)typical | TTL | Broadcast per key | Dropped message: silent staleness until TTL |
| Delete on write (at-least-once)protocol | TTL | Durable topic | Fill race: a slow reader re-caches the old value |
| Versioned keystypical | 0 for new readers | None (version must be read) | Garbage accumulation; a version lookup per read |
| Write-through with CASassumption | ~0 | None | Needs versioned values and conditional writes |
Practical mitigations, in the order worth applying them
Delete, never set, from the writer. A writer that sets a value into the cache is racing every other writer and every in-flight fill. A delete is idempotent, order-insensitive against other deletes, and the worst outcome is a miss. This one rule removes a whole class of bugs at the cost of one extra origin read.
Make fills conditional. Have the reader write back with a check: SET k = v1 IF absent or IF version < mine. The slow reader in the race above then fails to install its stale value. Redis-style SET NX gets most of this; a version comparison gets all of it.
Bound the damage with a short TTL. Every strategy above still has TTL as its backstop, so the TTL is not a tuning knob, it is the correctness bound. Pick it as "how long am I willing to be wrong in the worst case", and pick it before optimising hit rate.
Reconcile, do not trust. For values where staleness has business consequences, sample cache entries against the source on a schedule and report divergence. This is the only mechanism that *detects* the failure at all, since every other signal looks healthy. A coherence bug that nobody can observe will live in your system for years.
And be explicit about what none of this gives you: read-your-writes across instances. A user who writes and is then routed to a different instance may read a stale local cache. If the product needs it, that requires a session-scoped mechanism — pin the session, or bypass the cache for a window after a write by that user. It cannot be fixed at the cache layer, which is why it belongs with Read-After-Write: Letting a User See Their Own Change and Session Guarantees: The Underrated Middle Ground rather than here.
1// WRITER: delete, never set. Idempotent, order-insensitive, worst case a miss.2async function updateProduct(id: string, patch: Patch) {3 const version = await db.update(id, patch) // version is monotonic per key4 await cache.del(`product:${id}`) // at-least-once is fine: deletes commute5 return version6}7 8// READER: the fill must be conditional on what it read, or a slow reader can9// install a value that was already superseded before it got here.10async function readProduct(id: string): Promise<Product> {11 const key = `product:${id}`12 const hit = await cache.get(key)13 if (hit) return hit.value14 15 const { value, version } = await db.getWithVersion(id)16 17 // Install only if nothing newer is present. Without this check, the18 // invalidation that fired while we were reading is simply overwritten.19 await cache.setIfVersionGreater(key, { value, version }, version, jitteredTtl())20 return value21}Key points
- A perfectly delivered invalidation can still leave a permanently stale entry, because it races an in-flight fill carrying an older value.
- Invalidation inherits its carrier’s delivery semantics: at-most-once loses entries silently, at-least-once is safe because deletes are idempotent.
- TTL is the only unconditional bound on staleness; invalidation reduces expected staleness, never the maximum.
- Delete-on-write plus conditional fills closes the race without any new messaging; versioned keys avoid invalidation entirely.
- No cache-layer mechanism gives read-your-writes across instances — that needs session pinning or a post-write bypass window.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A writer updates the source and produces a monotonic version for the key.
- • The writer deletes the cache entry rather than setting it, so the operation is idempotent and commutes with other deletes.
- • A reader that misses fetches the value together with its version from the source.
- • The reader installs the value conditionally — only if absent or if its version is newer than what is present.
- • TTL remains as the unconditional backstop, and a reconciliation job samples entries against the source to detect divergence that all other signals hide.
- • An invalidation message is dropped and the entry stays stale until TTL, with no error anywhere.
- • A slow reader re-caches a value it read before the write, resetting the TTL on stale data.
- • Invalidations for one key are processed out of order, so an older delete arrives after a newer fill.
- • One instance in the fleet is not subscribed — a rolling deploy, a network policy, a crashed consumer — and serves stale values indefinitely.
- • A bulk source update produces more invalidations than the messaging layer can deliver, and the backlog grows past the TTL.
- • Permanently stale key: a specific product shows the old price for hours while every other key is correct. Invalidation metrics show successful delivery, and only a manual cache inspection reveals it.
- • Divergence across instances: repeated identical requests alternate between old and new values as the load balancer rotates instances. No error, no pattern in the logs, and it disappears when you look at one instance.
- • Silent subscriber loss: one pod stops consuming the invalidation topic and serves stale data for its whole lifetime. Consumer lag on that partition is the only signal, and only if per-consumer lag is exported.
- • Post-write staleness reported by users: someone edits their profile, reloads, and sees the old value roughly half the time — instance-local caches with no read-your-writes handling.
- • TTL-only caching needs no coordination at all and is the baseline every other strategy is measured against.
- • Invalidation is one-way messaging — cheaper than consensus, and it buys expected freshness rather than any guarantee.
- • Conditional fills need a compare-and-set primitive at the cache, which is a small piece of coordination *at the copy* and closes the race that messaging alone cannot.
- • Genuine coherence — every reader always sees the latest write — requires reading through a single authority on every request, which is to say not caching at all.
- • If invalidation delivery is broken, staleness silently reverts to the TTL bound. The system keeps serving, incorrectly, with no degradation signal.
- • Under a partition between writer and cache, writes succeed at the source and caches serve old values — availability preserved, coherence abandoned.
- • Deletes being idempotent means duplicate invalidations are harmless, so retries and at-least-once delivery cost nothing but bandwidth.
- • Detect: sample cache entries against the source and report a divergence rate. This is the only detector; every other signal is healthy during a coherence bug.
- • Contain: keep the TTL short enough that the worst case is tolerable, and treat that number as a correctness parameter rather than a performance one.
- • Recover: flush the affected key space — cautiously, since a broad flush hands the origin the full multiplier described in A Cache Across Machines Is a Replica With No Replication Protocol.
- • Reconcile: for values with business impact, reconcile the derived copies against the source on a schedule and alert on the delta rather than on delivery success.
- • Verify: run the fill race deliberately in a test — read, delay, write, invalidate, complete the fill — and confirm the cache ends with the new value.
- • Cache-versus-source divergence rate from periodic sampling, which is the only direct measure of coherence.
- • Invalidation delivery lag and per-consumer lag, so a single instance that has stopped consuming is visible.
- • Age of served entries as a distribution, which shows whether TTL is actually the binding bound or whether entries are being refreshed early.
- • Count of conditional fills rejected because a newer version was present — a direct count of races caught rather than lost.
- • Read-heavy data with a clear owner that can emit invalidations, where TTL alone would be too stale to be useful.
- • Values whose freshness matters within seconds but not within milliseconds — the regime where invalidation genuinely pays for itself.
- • Systems with a shared cache tier, where one invalidation covers all readers and the fan-out problem does not arise.
- • Rapidly changing data, where invalidation traffic approaches write traffic and the cache is mostly cold anyway.
- • Large per-instance cache fleets, where every invalidation is a broadcast and the messaging cost scales with instance count.
- • When invalidation is used to justify a long TTL: the TTL is the bound, and stretching it because "invalidation will handle it" is how multi-hour staleness bugs are born.
- • TTL only, with a TTL short enough to be acceptable. No messaging, no races, and much easier to reason about than anything else here.
- • Versioned keys, which eliminate invalidation entirely by never reusing a key after a change. Costs a version read and some garbage.
- • Read through a single shared cache with write-through updates, trading a network hop for the disappearance of cross-instance divergence.
- • Do not cache mutable data; cache immutable derivatives keyed by content hash, which cannot become stale by construction.
The race that produces permanent staleness
What people believe, and what is true
We publish invalidations, so the cache is coherent.
Invalidation lowers expected staleness. The maximum is still the TTL, and a fill racing an invalidation can install a stale value with a fresh TTL despite perfect delivery.
The writer should update the cache with the new value.
Setting from the writer races other writers and in-flight fills. Deleting is idempotent and order-insensitive, and the worst case is one extra origin read.
Duplicate invalidations are a problem.
For a plain delete they are harmless, which is exactly why at-least-once delivery is the right choice here.
Caching gives read-your-writes as long as we invalidate.
Not across instances. A user routed to a different instance can read that instance’s stale local copy; you need session pinning or a post-write bypass window.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
When data changes, every cached copy is wrong until something tells it. That something is a message — and messages can be lost, arrive late, or land before the copy they were meant to remove even exists.
Practical
Delete from the writer rather than setting, make reader fills conditional on version, deliver invalidations at-least-once because deletes are idempotent, and treat the TTL as your correctness bound rather than a tuning knob. Then add sampled reconciliation, because nothing else detects a coherence bug.
Advanced
The fill race is a lost-update problem between a read-modify-write on the cache and a concurrent source write, and it has the standard solution: make the write conditional on the version observed. That reframing tells you exactly what a coherence protocol needs — a per-key version that is monotonic at the source and carried into the cache — and it explains why versioned keys work so well, since a key that is never reused after a change cannot lose an update.
Apply it
- 💬 Walk through how a correctly delivered invalidation can still leave a permanently stale cache entry.
- 💬 Why should a writer delete the cache entry rather than update it?
- 💬 How would you even detect that a cache has gone incoherent, given that every metric looks healthy?