advanced · investigation

The Webhook That Paid Twice

An order was marked paid twice, releasing duplicate store credit to the customer. Finance reconciliation later flags several orders whose paid events outnumber provider charges.

Evidence

webhook consumer log (order 7731):
  09:12:03 POST /hooks/payments  event_id=evt_881 delivery_id=d_1 sig=valid  -> 200 (credit issued)
  09:12:19 POST /hooks/payments  event_id=evt_881 delivery_id=d_2 sig=valid  -> 200 (credit issued AGAIN)

handler: verify HMAC -> parse -> mark_paid(order_id) -> issue_credit(order_id)
         no event_id lookup anywhere; mark_paid is a blind UPDATE, not conditional
sig scheme: HMAC(raw_body) only — no timestamp in signed material, no replay window
provider docs: "delivery is at-least-once; deduplicate on event_id"

Investigate

Inspect Signature verification

HMAC over the raw body verifies correctly — the duplicate is authentic, re-sent by the provider after the first 200 was slow to arrive. Signatures authenticate origin; they say nothing about “only once”.

Inspect Consumer idempotency

The handler never records processed event_ids: `evt_881` is processed fresh both times, and `issue_credit` is not idempotent, so side effects double.

Inspect Replay protection

The signed material contains no timestamp and the handler enforces no freshness window — any captured delivery, or a provider redrive weeks later, replays with a perfectly valid signature.

Inspect State transition design

`mark_paid` is an unconditional update rather than a pending→paid transition, so even the second processing had nothing to bounce off.