intermediate

Checkout is slow and every dashboard is green

p99 checkout latency hits 6 s while CPU, memory and database all look healthy.

The page

SLO BURN · checkout-api · latency SLO (99% < 800 ms) burning at 14× · 2h budget remaining · p99 = 5,940 ms

Timeline — in the order it was observed

Observation order is not causal order. The first thing anyone noticed is rarely the first thing that happened.

13:32checkout-api v3.11 deployed. Routine release: a copy change and a feature flag, no code on the payment path.deploy marker
14:07Support escalates three customer reports of checkout "hanging" and then eventually working.support queue
14:12SLO burn-rate alert fires for checkout-api latency. p99 is 5,940 ms against an 800 ms objective.slo_burn_rate
14:14On-call opens the service dashboard. CPU 30%, memory 41%, no errors, no restarts. Everything is green except latency.host metrics
14:16Someone proposes rolling back v3.11 on the grounds that it is the only thing that changed today.incident channel
14:19A captured trace for a slow checkout shows a single 5.2 s span inside a 5.9 s request.trace waterfall
14:23The payment provider status page reads "All Systems Operational". Nobody has looked at our own measurement of their latency.vendor status page
14:31Checkout conversion is down 22% for the hour. Requests are completing, just slowly enough that customers abandon.business metrics

The system

Pull up evidence · 0/7 opened

Most of this evidence is consistent with several explanations. Keep going until something narrows it.