Caching Architecture
Caches sit at five distances from the user — browser, CDN, in-process, distributed, database buffer — each one trading freshness for latency and load, and the architecture questions are which layer answers which read, how a key expires, and what happens when thousands of requests miss at once.
Most reads ask for the same few things repeatedly, and every one of them that reaches the database costs CPU, I/O and milliseconds. Caches answer repeated reads closer to the user, cutting p99 and shielding the database from load it would otherwise fall over under.
Five layers, five distances
A read can be answered in the browser (0 ms, no request), at the CDN edge (~10–30 ms, no origin load), from an in-process cache inside the application instance (microseconds, but per instance), from a distributed cache such as Redis shared by all instances (~0.5 ms, one round trip), or from the database buffer pool (still a query, but no disk). Each layer is further from the user, more consistent across instances, and more expensive per hit than the one before it.
The architecture decision is which reads belong at which layer. Static assets: browser and CDN. Anonymous pages: CDN. Per-user data: Redis, keyed by user. Configuration and feature flags: in-process with a short TTL. Everything else: the database, which caches its own hot pages. The patterns for filling and writing each cache — cache-aside, read-through, write-through, write-behind — are in Caching Patterns.
HTTP caching: Cache-Control and ETag
The browser and the CDN speak the same protocol. Cache-Control: max-age=N says "reuse this for N seconds without asking". public lets shared caches (the CDN) store it; private restricts it to the browser — a per-user page must be private or the CDN will serve one user’s account page to another. immutable with a hashed filename lets assets cache for a year with no revalidation. An ETag is a version stamp; when max-age expires the client sends If-None-Match and the server answers 304 Not Modified with no body, which turns a 200 KB response into a 100-byte round trip.
stale-while-revalidate=30 lets the cache serve the expired copy immediately and refresh in the background — the single cheapest p99 win for content that can be 30 s old.
# hashed static asset: cache forever, change the URL to change the file Cache-Control: public, max-age=31536000, immutable # anonymous product page: CDN may store it, serve stale briefly while refreshing Cache-Control: public, max-age=60, stale-while-revalidate=300 ETag: "a1b2c3" # per-user account page: browser only, always revalidate Cache-Control: private, no-cache ETag: "9f8e7d"
In-process vs distributed
An in-process cache is a Hash Map with an eviction policy — an LRU Cache or LFU Cache bounded by entry count or bytes. It is the fastest cache possible and the least consistent: twenty instances hold twenty copies that expire at twenty different moments, and an invalidation must reach all of them (usually by a Redis pub/sub message, or by simply keeping the TTL short). A distributed cache holds one copy for everyone, so invalidation is one DEL, at the cost of a network hop and a component that can be down.
Use both as levels: in-process for the few hundred hottest keys with a 5–30 s TTL, Redis as the shared second level, database last. The in-process level absorbs the hot keys that would otherwise make one Redis shard hot.
| Property | In-process (LRU map) | Distributed (Redis) |
|---|---|---|
| Hit latency | ~1 µs | ~0.5 ms |
| Copies | One per instance | One |
| Invalidation | Broadcast or short TTL | Single DEL |
| Survives a deploy | No — cold start | Yes |
| Memory | Competes with the app heap | Dedicated, sized independently |
| Failure | Cannot fail separately | Down → every read misses |
Stampedes, hot keys and negative caching
A cache stampede is what happens when a popular key expires and 3,000 concurrent requests all miss, all query the database for the same row, and all write the same value back. The database sees 3,000 identical queries in 50 ms. Worse is the version in the challenge midnight-cache-stampede: every key was written with the same TTL at deploy time, so they all expire in the same second. Three fixes stack: jitter the TTL (ttl + random(0, ttl × 0.1)) so expiries spread; single-flight so only one request per key recomputes and the rest wait on it; early recompute so a request that sees a key within its last few seconds refreshes it in the background while still serving the current value.
A hot key is one key read 100,000 times a second — a celebrity profile, a homepage config. Redis serves it from one shard, so that shard saturates while the others idle; replicate hot keys into the in-process level or spread them across N copies (key:0..N-1). Negative caching stores the fact that something does not exist (user:123 → NOT_FOUND, TTL 30 s) so a scraper hitting nonexistent ids cannot turn every miss into a database query; a Bloom Filter in front of the cache answers "definitely not there" without any storage per key.
1const inflight = new Map<string, Promise<string>>()2 3async function cached(key: string, ttl: number, load: () => Promise<string>) {4 const hit = await redis.get(key)5 if (hit) {6 const left = await redis.ttl(key)7 if (left < ttl * 0.1) void refresh(key, ttl, load) // early recompute in the background8 return hit // serve the current value now9 }10 const running = inflight.get(key) ?? refresh(key, ttl, load) // single-flight per key11 inflight.set(key, running)12 try { return await running } finally { inflight.delete(key) }13}14 15async function refresh(key: string, ttl: number, load: () => Promise<string>) {16 const value = await load()17 const jitter = Math.floor(Math.random() * ttl * 0.1)18 await redis.set(key, value, 'EX', ttl + jitter)19 return value20}Invalidation: the part that decides whether the cache is correct
A cache is a copy, and every copy can be stale. TTL bounds staleness to N seconds and needs no coordination; it is right for anything where "up to a minute old" is acceptable. Explicit invalidation — DEL on write — gives freshness at the cost of every writer knowing every key that depends on the row, which is where bugs live: the product cache is cleared but the category listing that embeds the product is not. Versioned keys sidestep it: bump catalog:v on any write and read product:{id}:{v}, so old entries are never read again and expire on their own. The full treatment, including write-around races where a slow reader writes a stale value after the invalidation, is in Cache Invalidation, Stampedes and Hot Keys.
Key points
- Browser → CDN → in-process → Redis → database buffer pool: each layer further, more consistent, more expensive per hit.
Cache-Controldecides who may store a response;ETagturns an expired entry into a 304 instead of a full body.- Two levels — in-process LRU for the hottest keys, Redis shared — absorb hot keys and survive deploys.
- Stampede defences stack: jittered TTL, single-flight per key, early recompute.
- Invalidation is the correctness problem: TTL bounds staleness, explicit DEL requires knowing every dependent key, versioned keys avoid both.
Browser → CDN → app cache → Redis → database
How data moves through it
One request or event, hop by hop.
- 1Browser → local cache: fresh
max-ageentry served with no request. - 2Browser → CDN: edge lookup by cache key; hit returns in ~20 ms,
If-None-Matchrevalidation returns 304. - 3CDN → app: miss forwarded to the origin; the app checks its in-process LRU.
- 4App → Redis:
GET product:42; hit returns the serialized value. - 5App → DB: on miss, the query runs; result written back with
SET … EX ttl+jitter. - 6DB → app → CDN → browser: response headers decide which layers store it on the way back.
When to use — and when not
- Reads outnumber writes and repeat: a product page read 10,000× per change, a feed read 50× per post.
- A measured database bottleneck on the read path, after indexes (Why Is This Query Slow? Indexes) and a connection pool are in place.
- Latency budgets that a database round trip cannot meet — p99 under 50 ms for a page that joins six tables.
- Data that must be strictly current for correctness — account balances, inventory at checkout, permissions — unless invalidation is explicit and tested.
- Write-heavy or unique-read workloads where the hit rate would be under ~50%; the cache adds a hop and a failure mode for nothing.
- Before the database has been measured; a missing index is a 100× win with no staleness.
Tradeoffs
Enormous read scaling and latency wins in exchange for staleness you must reason about per key and a component whose loss turns every read into a database query.
How it fails
- Stampede: a hot key expires and thousands of identical queries hit the database within milliseconds; with a shared deploy-time TTL, every key expires at once.
- Stale reads after a write because the writer did not invalidate the dependent listing or the second-level cache on other instances.
- A per-user response marked
publicand cached by the CDN: one user sees another’s data. - Hot key saturating one Redis shard while the cluster is 90% idle.
- Cache node loss with a 95% hit rate: the database receives 20× its normal load and falls over — the cache was load-bearing without anyone deciding it should be.
How it scales
- Hit rate is the lever: from 90% to 99% cuts database reads by 10×; measure it per key class, not globally.
- Redis scales by clustering on key hash; hot keys need local caching or replicated copies because a single key cannot be split.
- CDN layer scales reads to near-infinite for anonymous content; the origin only ever sees misses.
- When the cache becomes load-bearing, size the database for a cache-miss storm or add a second Redis replica so it never fully disappears.
How it interacts with databases, queues, caches, APIs and external systems
- Database: the source of truth behind every miss; it must survive the cache disappearing.
- Cache (Redis): shared level with TTL, atomic
SET NXfor locks, pub/sub for cross-instance invalidation. - CDN: shared HTTP cache driven by
Cache-Control; see CDN Architecture for keys and purge. - Queue: write events can drive invalidation asynchronously when the writer cannot know every dependent key.
- External API: responses from rate-limited third parties are cached to stay under their quota, with negative caching for their errors.