Hot Keys: When Aggregate Metrics Hide a Saturated Node
Sharding distributes keys, not traffic. One product goes viral, forty percent of requests land on one key, and the node holding it saturates while the cluster reports comfortable average utilization across every other node.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Distribution guarantees are about keys, not requests
Consistent hashing and range partitioning both distribute the *keyspace* evenly. Neither promises anything about how traffic distributes across that keyspace, because traffic distribution is a property of user behavior. When one product trends, one celebrity posts, or one tenant is a hundred times larger than the rest, request volume concentrates on a small number of keys and the node holding them absorbs a share of traffic wildly disproportionate to its share of keys.
The arithmetic is unforgiving. With ten nodes and 40% of requests targeting one key, that key's node receives 40% of all traffic while the other nine share 60% — roughly 6.7% each. The hot node is running at six times the fleet average. Cluster-average CPU reports something around 25% and every capacity dashboard reads green while one node drops requests.
Read the panel below for the general lesson about aggregates: an average over units with different loads answers a question nobody asked. The useful readings are the maximum, the ratio of maximum to median, and the distribution itself. Any metric that can be skewed should be published as a distribution, not a mean — the same argument Percentiles: Which One, and How Many Users Is That? makes about latency, applied to nodes instead of requests.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Cluster average CPU | 25% | Comfortable. This is the number on the capacity dashboard. | normal |
| Max node CPU | 98% | One node is saturated. Six times the fleet average. | smoking gun |
| Max-to-median node request rate | 6.1x | The skew ratio. Any value well above 1 means aggregates are lying. | smoking gun |
| Top key share of requests | 40% | One key. Sharding cannot help — the key is atomic. | smoking gun |
| p50 latency | 4 ms | Requests to the nine healthy nodes are fine. | normal |
| p99 latency | 780 ms | Requests to the hot node queue. The tail is entirely one node. | suspect |
| Cluster memory used | 38% | Not a capacity problem in any aggregate sense. | normal |
Detecting skew before it is an incident
Per-key metrics are the obvious answer and the wrong one at full fidelity: emitting a time series per cache key is the textbook cardinality explosion that takes down the monitoring system (Cardinality: The Label That Took Down Monitoring). The practical approach is bounded. Track per-node rates always, since node count is small and stable, and compute max-to-median as a single skew series — that one number detects the condition without naming the key.
Then, to identify *which* key, sample rather than enumerate: a top-N heavy-hitter sketch (count-min sketch or a simple periodic top-K over a sampled request stream) gives the hot keys at bounded cost. Many caches offer this natively — Redis has key-pattern statistics and slow-log facilities that surface hot patterns without per-key metrics. The DSA machinery underneath is a bounded Min-Heap over a streaming frequency map (Sliding Window with Frequency Map), which is a nice reminder that the algorithms domain is directly load-bearing in operations.
The strongest early-warning signal is the skew ratio trending upward over days. Traffic concentration rarely appears instantly; it usually builds as a product trends or a tenant grows. A max-to-median ratio climbing from 1.2 to 3.0 over a week is a capacity conversation with time to plan, while the same ratio discovered at 6.1 during an incident is an emergency.
| Mitigation | How it works | Works when | What it costs |
|---|---|---|---|
| Local (in-process) cache | Each application instance caches the hot value for a short TTL | Read-heavy, brief staleness acceptable | Staleness bounded by TTL; memory on every instance |
| Replicate the hot key | Store copies as key#1..key#N, read a random one | Read-heavy, many nodes available | Writes must fan out to all copies; consistency window |
| Split the key | Shard a counter or list into N sub-keys, combine on read | Aggregatable values (counters, sets) | Reads get more expensive; N must be maintained |
| Request coalescing at the edge | Collapse concurrent identical requests before they reach the store | Bursty read patterns | Adds a layer; same trade as Cache Stampede: Everyone Misses at Once coalescing |
| Dedicated capacity for the hot tenant | Route the outsized tenant to its own nodes | Multi-tenant skew rather than single-key skew | Operational complexity; routing rules to maintain |
| Rarely works | Add more nodes | Nothing — the key still lives on one node | Money; the hot node remains hot |
| Rarely works | Change the hash function | Redistributes keys, not the traffic to one key | A rebalance, for no benefit |
Why adding nodes does not help, and what does
The instinct on seeing a saturated node is to add capacity, and it fails here for a structural reason worth internalizing: a key lives on exactly one node, so scaling the cluster changes which node is hot without changing that a node is hot. Ten nodes to twenty halves each node's share of the *keyspace* and leaves the hot key exactly where it was, on one machine, receiving 40% of traffic. The bill doubles and the p99 does not move.
What works is one of two things: stop sending the requests to that node, or stop the key from being atomic. Local caching in each application instance is the first and is usually the fastest win for read-heavy hot keys — a one-second TTL on a value read ten thousand times per second reduces the store's load by roughly a factor of the instance count times the TTL in seconds, at the cost of up to one second of staleness. Replication of the hot key across N copies with random read selection achieves the same thing inside the store, and moves the cost to write fan-out.
For aggregatable values, key splitting is the durable answer and it is the same technique as sharding a contended counter in Low CPU, High Latency: Lock Contention — the identical structural problem, appearing in the cache layer rather than the database layer. One contended thing becomes N less-contended things, reads pay a combine cost, and the concurrency ceiling rises by roughly N. When neither applies — a single large object that must be strongly consistent and is read enormously — the honest answer is that the access pattern needs to change, which is a product conversation rather than an infrastructure one.
Key points
- Sharding distributes the keyspace evenly; traffic distribution is a property of user behavior and is frequently extreme.
- Cluster-average utilization averages a saturated node with idle ones and produces a number that describes nothing real.
- Track max-to-median node request rate as a standing skew metric — it detects the condition without per-key cardinality.
- Adding nodes does not help: the hot key still lives on one node, so the fleet grows and the hot node stays hot.
- Fixes either stop requests reaching the node (local caching, edge coalescing) or stop the key being atomic (replication, splitting).
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Users → symptom: intermittent slow page loads; p50 is 4 ms and p99 is 780 ms, so most requests are fine and some are terrible.
- 2Cluster → false comfort: average CPU 25%, memory 38% — no aggregate signal indicates a problem.
- 3Per-node → localization: node 3 sits at 98% CPU while the other nine average 7%; the tail is entirely requests routed to node 3.
- 4Per-key → cause: one product key accounts for 40% of all requests after a social media post, and consistent hashing places it on node 3.
- 5Cause → root cause: traffic skew against an evenly distributed keyspace, so adding nodes cannot help — the key is atomic and lives in one place.
- • "Average CPU is 25%, we have plenty of headroom." One node is at 98% and dropping requests.
- • "Add nodes to spread the load." The hot key lives on one node regardless of fleet size.
- • "The hash function is unbalanced." It balanced the keyspace correctly; traffic is what is unbalanced.
- • "p50 is fine, so most users are fine." Every user who touches the hot key is affected, and that is 40% of requests.
- • "Rebalance the cluster." Rebalancing moves the key to a different node, which then becomes the hot one.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Per-node request rate and CPU, published as max and median rather than as a fleet average.
- • Max-to-median ratio as a single skew series, alerted on when it trends upward over days.
- • Top-N hot keys from a sampled heavy-hitter sketch or the store's native key statistics — bounded cost, no per-key time series.
- • p99 latency split by node, which localizes the tail to one machine immediately.
- • Per-tenant request share in multi-tenant systems, where skew is usually a tenant rather than a key.
- • Add a short-TTL in-process cache in front of the store for read-heavy hot keys — the fastest win, bounded staleness, no store changes.
- • Replicate the hot key across N copies read at random, moving the cost to write fan-out and spreading reads across the fleet.
- • Split aggregatable values into N sub-keys combined on read — the same technique as sharding a contended counter in [[lock-contention]].
- • Coalesce concurrent identical requests at the edge so a burst becomes one fetch, as in [[cache-stampede]].
- • Give an outsized tenant dedicated capacity when the skew is per-tenant rather than per-key, and route explicitly.
- • Max-to-median node request rate falls toward 1, measured at the same total traffic level.
- • p99 latency converges toward p50 — that gap was the hot node, so closing it is the proof.
- • Confirm the previously hot node's CPU dropped and no other node rose to take its place, which would mean the key simply moved.
- • For local caching, measure the actual staleness users see against the freshness contract, not just the load reduction.
- • Local caching bounds staleness by TTL and multiplies memory use across every application instance.
- • Key replication spreads reads and makes every write fan out to N copies, with a consistency window between them.
- • Key splitting raises the concurrency ceiling and makes reads more expensive, with N as a parameter that must be revisited as traffic changes.
- • Dedicated tenant capacity isolates the problem and fragments the fleet, leaving idle capacity stranded per tenant.
- • A standing alert on the skew ratio, which gives days of warning as concentration builds rather than minutes during an incident.
- • Per-node dashboards that show max and median by default, with fleet averages removed from capacity views entirely.
- • A load test with a deliberately skewed key distribution, asserting the system degrades gracefully rather than pinning one node.
- • A periodic review of top-N keys, so a new hot key is noticed by a process rather than by an outage.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe ten-node cluster, 40% key share and 6.1x skew ratio are constructed to make the arithmetic legible. Real skew distributions are usually Zipf-like with a long tail rather than one dominant key.
- ENVIRONMENT-SPECIFICWhether a hot key saturates a node depends on node capacity, request cost and whether the store is single-threaded per shard — Redis being effectively single-threaded per instance makes hot keys bite considerably harder.
- WORKLOAD-SPECIFICLocal caching effectiveness depends on read/write ratio and acceptable staleness; it does nothing for a write-hot key.