Async ProcessingdebuggingIntermediate

The customer was charged twice after a 30-second timeout

Symptoms

  • Support tickets: customers see two identical card charges for one order, a few seconds apart.
  • The mobile app showed a spinner for 30 s, then "Something went wrong — please try again". Tapping again "worked".
  • It affects ~0.3% of checkouts, and only during the evening peak (19:00–21:00).
  • The payments table has two rows for the same order_id, each with a different provider charge id.
# api-gateway   (upstream timeout: 30 000 ms)
19:42:07.114  req=7f3a  POST /orders/8841/pay  → payment-svc
19:42:37.120  req=7f3a  504 upstream timeout after 30000 ms
19:42:38.905  req=9c21  POST /orders/8841/pay  → payment-svc     (same device, automatic retry)
19:42:41.330  req=9c21  200 OK  (3.4 s)

# payment-svc
19:42:07.118  req=7f3a  order=8841  provider.charge(amount=4990) start
19:42:38.910  req=9c21  order=8841  provider.charge(amount=4990) start
19:42:39.671  req=7f3a  order=8841  provider.charge ok  psp_id=ch_1Q8x   (32.5 s)
19:42:39.680  req=7f3a  order=8841  INSERT INTO payments ... status=captured
19:42:41.325  req=9c21  order=8841  provider.charge ok  psp_id=ch_1Q8y   (2.4 s)
19:42:41.328  req=9c21  order=8841  INSERT INTO payments ... status=captured

# payments
id    order_id  psp_id    amount  created_at
501   8841      ch_1Q8x   4990    19:42:39
502   8841      ch_1Q8y   4990    19:42:41

Investigate

Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.

Payment provider status
Recent deploys
Gateway and service timeouts
Mobile client retry logic
The call to the provider
Constraints on the payments table
Database locks and CPU