Local vs Distributed Cache
In-process is faster and per-instance; shared is consistent and one more thing that can be down. The choice is about invalidation, not speed.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
Should the cache live inside the process or in a shared store, and what does each choice do to correctness?
Feature-flag evaluation and permission lookups happen on every request. They should not cost a network round trip each time.
A module-level Map in the process. No network hop, no serialization, no new dependency to operate. It is obviously the fastest option.
It is the fastest option and it is N caches, not one. With ten instances behind a load balancer, ten copies diverge independently, and a user refreshing the page sees the new value and then the old one depending on which instance answered (Sticky Sessions).
- It is the fastest option and it is N caches, not one. With ten instances behind a load balancer, ten copies diverge independently, and a user refreshing the page sees the new value and then the old one depending on which instance answered (Sticky Sessions).
- Invalidation reaches one instance. The write handler that runs on instance 3 clears instance 3's map; instances 1, 2 and 4 through 10 keep serving the old value until their TTLs expire.
- An unbounded map is a memory leak with a friendly name. Without a size limit and an eviction policy it grows until the process is OOM-killed, and it will look like a leak rather than a cache (Memory Leaks in Backend Services).
- Every deploy, restart and scale-out event starts with an empty cache on the new instance, so cold-start misses are routine rather than exceptional (Cache Warmth and the Real Cost of Migration).
- It disappears with the process, so nothing you cannot recompute may live there. Storing session state in it means a deploy logs everyone out (Stateless Services).
What is actually happening
- A local cache is memory in your process: a hash map with a size bound and an eviction policy. Lookup is a pointer dereference — no network, no serialization, no connection pool. Its scope is one process and its lifetime is that process (Hash Map).
- A distributed cache is a separate service reached over the network. Every read costs a round trip plus serialization on both ends, and it is shared: one populate serves the whole fleet, one delete invalidates for the whole fleet.
- The real axis is not latency, it is invalidation scope. Local caching means an invalidation event has to reach N processes; shared caching means it has to reach one place. Everything else follows from that.
- A two-tier cache combines them: local in front, shared behind. The local tier absorbs the hot keys, the shared tier absorbs local misses and keeps the fleet roughly aligned. The complexity is that the local tier needs its own, much shorter TTL, and invalidation must now clear both.
- Broadcast invalidation (pub/sub to every instance) makes local caches invalidatable fleet-wide. It is best-effort: an instance that was restarting, partitioned or slow when the message went out simply misses it, so a TTL is still mandatory underneath.
- A local cache is process state; a distributed cache holds application state. Confusing the two is how "restart the service" becomes a data-loss event (Stateless Services).
The same picture, twice
Drawing both makes the real difference obvious, and it is not the network hop. In the local picture there are three caches and no way to reach all three from a write. In the shared picture there is one cache, one place to invalidate, and one thing that can be unavailable.
Everything in this lesson follows from those two shapes. If per-instance divergence is acceptable for this data, take the fast one. If it is not, you are buying a dependency, and you should decide now what happens when it is down.
Choosing by what the data tolerates
The question that resolves this in one step is: if two users hit two different instances one second apart, is it acceptable for them to see different values? For a feature-flag snapshot, yes. For a shopping-cart total, no. For a permission set, only if your revocation window can absorb it.
Notice that the answer for a given piece of data is stable across scale. What changes with scale is how many divergent copies exist, not whether divergence is tolerable.
Can two instances legitimately disagree about this value?
when Effectively immutable within the TTL: parsed config, compiled templates, reference data, feature-flag snapshots. Divergence for a few seconds is invisible.
cost N divergent copies, no reliable invalidation, cold start on every deploy and scale-out, memory per instance.
when Every caller must see the same value; invalidation must be immediate fleet-wide; the value is expensive enough that per-instance populates are wasteful.
cost A network round trip per read, serialization both ways, a service to operate, and a new dependency on the critical path.
when A small number of very hot keys where even one shared round trip per request is significant.
cost Two TTLs, two invalidation paths, two hit-rate metrics, and the local tier can still be stale after the shared tier is corrected.
when Local speed is needed but divergence must be short — permission or flag data with a tight revocation requirement.
cost A pub/sub dependency, best-effort delivery, and a TTL still required underneath for instances that miss the message.
when The read is a single indexed lookup, or the data is written as often as it is read.
cost None — and this is the right answer more often than the module implies (When Not to Cache).
What each one is actually made of
Comparing the two on properties rather than on speed makes the decision mechanical. The rows people underweight are lifetime and failure impact: a local cache disappears on every deploy, and a shared cache is a service that can be down while your service is up.
The last row is the one to plan for before it happens. "The cache is down" is a capacity event with a known shape, and the mitigation — a bounded fallback, a circuit breaker, a degraded response — has to exist before the day it is needed (Circuit Breakers).
| Property | Local (in-process) | Distributed (shared) |
|---|---|---|
| Lookup path | A pointer dereference in the heap | Network round trip plus serialize/deserialize |
| Scope | One process — N copies across the fleet | One copy for everyone |
| Invalidation | Reaches this process only, unless you broadcast | One delete, fleet-wide |
| Lifetime | Dies with the process: every deploy and scale-out is a cold start | Survives your deploys; may not survive its own failover |
| Memory | Your heap — competes with request handling, and bounds GC pressure | Its own, sized independently (Garbage Collection: Pause, Throughput, Footprint — Pick Two) |
| Capacity | Bounded by instance memory | Bounded by the cache cluster, scaled separately |
| New failure domain | None | Yes — it can be down, slow, full or failing over |
| When it is unavailable | Not possible; it is your heap | Full read load hits the origin at once (Cascading Failure) |
How to build it
Most important first.
- Use a local cache for data that is effectively immutable within its TTL and where per-instance divergence is acceptable: parsed configuration, compiled templates, feature-flag snapshots, reference tables.
- Use a distributed cache when the value must be the same for every caller, when invalidation must be immediate fleet-wide, or when populating is expensive enough that doing it once per instance is wasteful.
- Always bound a local cache: a maximum entry count, an eviction policy and a TTL. An unbounded map in a long-lived process is a leak (Resource Limits).
- For a two-tier cache, keep the local TTL short — seconds — and let the shared tier hold the longer window. The local tier is there to absorb hot keys, not to be the source of truth.
- Add broadcast invalidation only when you can also state what happens to an instance that misses the broadcast. The answer must be "its TTL expires shortly", which means the TTL is still the real bound.
- Never put anything in a local cache that cannot be recomputed from an authoritative source. The process will end at a time unrelated to your data (Graceful Shutdown).
What can go wrong
- Divergence across instances producing "it works on some requests" bug reports that are impossible to reproduce from one client.
- Unbounded growth to OOM, misdiagnosed as a leak because the memory is genuinely reachable (Leak or Unbounded Cache? The Question That Picks the Fix).
- The distributed cache becoming a hard dependency: when it is down, every request falls through to a database sized for cached traffic (Cascading Failure).
- Two-tier caches where invalidation clears the shared tier only, so local copies persist and the bug appears fixed on some instances.
- Broadcast invalidation lost during a rolling deploy — the instance was not listening yet — leaving one instance stale for a full TTL.
- Serialization cost in a distributed cache exceeding the query it replaced, for large values.
- A distributed cache node failover emptying the dataset instantly: a fleet-wide cold start, which is the maximum-severity stampede (Cache Stampede).
- Concurrent read-modify-write of a cached counter: two instances read the same value, both increment, one write is lost. Use the cache's atomic operations rather than get-then-set (Atomic Operations).
- Invalidation racing a populate across instances, so one instance repopulates from a stale read after the broadcast has already passed it by.
- Two-tier populate ordering: the local tier is populated from a shared value that is invalidated microseconds later, leaving a stale local copy for the local TTL.
- A local single-flight map mutated concurrently on a threaded runtime, needing an atomic get-or-insert rather than check-then-set (Double-Checked Locking: The Canonical Cautionary Tale).
- A shared cache is a shared trust boundary. Every service with credentials to it can read every key, so a compromise of the least-important service reads the most-important service's cached data (Defence in Depth).
- Namespace keys per service and per tenant, and require authentication on the cache. Cache servers deployed with no auth inside a "private" network are a standard finding (Multi-Tenant Isolation).
- A local cache of authorization decisions must be invalidatable, or revocation has an N-instance-TTL-shaped delay you cannot shorten during an incident (Role-Based Access Control).
- Encrypt in transit to a distributed cache if it crosses any boundary you do not fully control. Cache protocols are typically plaintext by default.
- Do not let a cache outage change your authorization answer. Falling open on a cache miss for a permission check is an availability decision with a security consequence.
- "Local is faster, so local is better." Local is faster per lookup and wrong more often. The decision is about invalidation scope; latency is the tiebreaker, not the criterion.
- "A distributed cache guarantees consistency." It guarantees one *copy*. Read-modify-write on a cached value still races, and a stale populate is stale for everyone at once (Backend Races).
- "An in-memory map is not really a cache, so it does not need a TTL or a bound." An unbounded map in a long-lived process is exactly a cache, with the eviction policy set to "never" (Memory Leaks in Backend Services).
- "Sticky sessions fix local-cache divergence." They hide it, by making one user consistently hit one stale instance. The data is still divergent and the next deploy redistributes everyone (Sticky Sessions).
- "If Redis is down we just read from the database." True, and that is a 100% cache-miss load spike arriving at once. Whether it is survivable is a capacity question you should answer before it happens.
Operating it
- Hit rate per instance for local caches. A fleet average hides an instance whose cache never warmed.
- Heap usage attributable to the cache, and entry count against the configured bound. Both should be flat once warm; growth means the bound is not doing its job.
- For a shared cache: connection count, connection errors and latency percentiles from the *client* side. Server-side metrics miss client pool exhaustion entirely (Connection Pool Saturation: Waiting in Front of an Idle Database).
- Broadcast invalidation delivery: messages published versus applied per instance. The gap is your divergence.
- For two-tier caches: hit rate at each tier separately. A local tier with a low hit rate is pure added latency and memory.
- Post-deploy miss rate. A spike after every deploy is the shape of local-cache cold start ("What Changed?" — Deploy Markers and the Invisible Deploys).
- Local caches scale beautifully in throughput — every instance added brings its own cache — and badly in consistency: divergence is proportional to instance count.
- A distributed cache is a shared resource with its own limits: connections, memory, network. Above a certain fleet size the connection count alone becomes a constraint, especially with serverless-style short-lived instances (Serverless and Database Connections).
- At 100x, two-tier becomes close to mandatory: the shared cache cannot absorb every read from every instance for the hottest keys (Hot Keys: When Aggregate Metrics Hide a Saturated Node).
- Aggressive autoscaling makes local caches worse, because instances are short-lived and never warm (Autoscaling a Backend).
- Multi-region turns a shared cache into a cross-region round trip. At that point each region needs its own, and they diverge from each other by design (Multi-Region Deployment).
- Local: no network hop, no new dependency, no serialization — paid for with N divergent copies and invalidation you cannot do reliably.
- Distributed: one consistent copy, fleet-wide invalidation, survives restarts — paid for with a network round trip on every read, a service to operate, and a new failure domain on the critical path.
- Two-tier: the best hit-rate profile, and two invalidation paths, two TTLs and two hit-rate metrics to reason about. Complexity roughly doubles.
- Broadcast invalidation makes local caches nearly consistent, at the cost of a pub/sub dependency and a best-effort delivery guarantee you must not treat as reliable.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe invalidation-scope trade holds for any runtime and any cache product.
- RUNTIME-SPECIFICWhat "in-process" means depends on the concurrency model. Node and Go share one heap per process, so a module-level map is shared by all in-flight requests. A pre-fork model (Gunicorn workers, PHP-FPM, Puma clusters) gives every worker its own memory, so a single machine already has as many divergent caches as it has workers — the divergence problem starts before you add a second server.
- CLOUD-SPECIFICServerless and short-lived container platforms make local caches close to useless: instances are frozen, recycled and scaled to zero on a schedule you do not control, so hit rate is a function of the platform's reuse policy rather than of your traffic (Serverless Trade-offs).
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.