Deadlines & Tail Latency
5 lessons. Every one names the guarantee it claims, what a node can know, and how it fails.
The client waits 2 seconds. A waits 2 seconds for B, B waits 2 seconds for C. Every value is defensible on its own and the composition is nonsense: by the time C is still working, the client left long ago and A and B are doing paid work for nobody.
Q · The client gives me 2 seconds. How much of it may each hop below me spend?
A budget nobody can see is not a budget. Unless the remaining time travels with the request, every service independently waits its own configured default — and work continues for a long time after the last person who cared has left.
Q · How does a service four hops down learn that it has 300ms left rather than its configured 30 seconds?
When nobody is waiting for a result, computing it is pure waste, and under load that waste is most of your capacity. But cancellation is a message that may not arrive, and stopping a write halfway through leaves state that matches no intention at all.
Q · My caller disconnected. Should I stop the work I am doing on their behalf — and can I safely?
Most slow responses are not slow because the work is hard — they are slow because that particular replica hit a garbage collection, a cold cache, or a noisy neighbour. Asking a second replica after the p95 has elapsed converts a tail problem into a small, bounded amount of extra load.
Q · One replica is slow for reasons that have nothing to do with my request. Can I just ask someone else?
Each shard is slow only 1% of the time, which sounds excellent. Fan a request out to 100 of them and the chance that at least one is slow is 63%. The aggregate does not inherit the component’s median — it inherits the component’s tail, amplified by the width of the fan-out.
Q · Every shard has a good p99. Why is the p50 of my fan-out request terrible?