Distributed SystemsdebuggingAdvanced
One cache node is at 95% CPU while the other three idle
Symptoms
- cache-c runs at 95% CPU and serves ~45% of all cache reads; cache-a, -b and -d sit at 10–25%.
- Only cache-c evicts keys; its hit rate is 71% against 96% on the others, so the database sees extra load for the keys that live there.
- p99 for keys on cache-c is 30 ms versus 1 ms elsewhere.
- Adding a fifth node (cache-e) last week barely helped: cache-c dropped from 45% to 44% of reads.
# client-side consistent hashing: ONE point per node, position = crc32(hostname) node point arc (to next point clockwise) share of keyspace cache-a 0x0A1F3C20 → 0x1C08B4E1 7.0 % cache-b 0x1C08B4E1 → 0x4D91E002 19.4 % cache-d 0x4D91E002 → 0x96000000 28.3 % cache-c 0x96000000 → wrap → 0x0A1F3C20 45.4 % ← owns nearly half the ring # after adding cache-e (point 0x05A00000, lands inside cache-c's arc, near its end) cache-e 0x05A00000 → 0x0A1F3C20 1.8 % cache-c 0x96000000 → wrap → 0x05A00000 43.6 % ← barely changed # per-node reads/s: a 9,800 · b 27,100 · d 39,600 · c 63,400 · e 2,500 # hottest single key: 0.4 % of reads — there is no hot key
Investigate
Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.
cache-c hardware
Hot keys
Ring construction
Adding cache-e
Client library versions
Traffic changes
Redis maxmemory on cache-c