← PracticeAdvancedFailure

The Config Push That Never Lands

Pull up the evidence one item at a time, commit to a diagnosis, and only then see the schedule that actually ran.

What was reported

Our config service holds a rate-table snapshot behind a read/write lock. Reads run at about 40,000 per second across 64 threads. A config push used to apply in under a millisecond. Since the traffic doubled last month, pushes sometimes take ninety seconds and sometimes never complete at all — the deploy tool times out at five minutes and reports a failed rollout, even though the service is serving reads perfectly the whole time. Restarting the pod applies the config instantly.
1std::shared_mutex rw;
2RateTable table;
3 
4Rate lookup(const Key& k) {              // 40,000/s across 64 threads
5  std::shared_lock<std::shared_mutex> g(rw);
6  return table.at(k);
7}
8 
9void publish(RateTable next) {           // a few times per day
10  std::unique_lock<std::shared_mutex> g(rw);
11  table = std::move(next);
12}

Evidence

Nothing here is labelled as relevant. Some of it is not.

What is actually happening?