Three retries at three layers finished off the payment provider

• The payment provider became slow (p99 from 180 ms to ~8 s). Four minutes later checkout availability was 2%.
• Outbound requests to the provider rose from ~420 rps to ~11,000 rps while checkout traffic stayed flat at ~400 rps.
• The provider returned 429, then temporarily blocked our API key for "abusive traffic".
• Recovery required turning checkout off entirely for ten minutes; turning it back on at full traffic immediately re-triggered the storm.

# rps per hop (1-minute averages)
time    checkout   gw→checkout   checkout→payment   payment→provider   provider p99
19:58   410        410           415                420                180 ms
20:00   408        612           1,540              3,900              2,100 ms
20:01   405        1,190         3,540              9,870              8,200 ms
20:02   402        1,206         3,610              10,480             timeout
20:03   398        1,210         3,630              10,900             429

# retry configuration, as deployed
gateway.yaml        upstream.max_attempts: 3   on: [5xx, timeout]   backoff: none   timeout: 10s
checkout-svc        http.maxAttempts: 3        on: [5xx, timeout]   backoff: none   timeout: 30s
payment-svc         provider.maxAttempts: 3    on: [5xx, timeout]   backoff: 100ms fixed   timeout: 60s
payment-svc         circuitBreaker.enabled: false     # "disabled after false trips, 2026-03"