Async ProcessingdebuggingIntermediate
The customer was charged twice after a 30-second timeout
Symptoms
- Support tickets: customers see two identical card charges for one order, a few seconds apart.
- The mobile app showed a spinner for 30 s, then "Something went wrong — please try again". Tapping again "worked".
- It affects ~0.3% of checkouts, and only during the evening peak (19:00–21:00).
- The payments table has two rows for the same
order_id, each with a different provider charge id.
# api-gateway (upstream timeout: 30 000 ms) 19:42:07.114 req=7f3a POST /orders/8841/pay → payment-svc 19:42:37.120 req=7f3a 504 upstream timeout after 30000 ms 19:42:38.905 req=9c21 POST /orders/8841/pay → payment-svc (same device, automatic retry) 19:42:41.330 req=9c21 200 OK (3.4 s) # payment-svc 19:42:07.118 req=7f3a order=8841 provider.charge(amount=4990) start 19:42:38.910 req=9c21 order=8841 provider.charge(amount=4990) start 19:42:39.671 req=7f3a order=8841 provider.charge ok psp_id=ch_1Q8x (32.5 s) 19:42:39.680 req=7f3a order=8841 INSERT INTO payments ... status=captured 19:42:41.325 req=9c21 order=8841 provider.charge ok psp_id=ch_1Q8y (2.4 s) 19:42:41.328 req=9c21 order=8841 INSERT INTO payments ... status=captured # payments id order_id psp_id amount created_at 501 8841 ch_1Q8x 4990 19:42:39 502 8841 ch_1Q8y 4990 19:42:41
Investigate
Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.
Payment provider status
Recent deploys
Gateway and service timeouts
Mobile client retry logic
The call to the provider
Constraints on the payments table
Database locks and CPU