Incidents Are Product Events
An incident is something that happened to customers, not to a server. Who was affected, what did they lose, what do we tell them and what do we owe them — the fix is one line of that response.
The ask, the obvious response, and how it goes wrong
Every lesson starts where the work starts: someone asked for something, and the first response that comes to mind has a problem.
When checkout breaks for forty minutes, what is the engineering team actually responsible for once the fix is deployed?
Friday, 16:40, in the incident channel: "Revert is out, order creation is healthy again. Can we close this?"
The incident is over when the system is healthy. Revert, confirm the graphs are green, write a short summary of the cause, close the ticket. Anything customer-facing is support's job.
Between 16:00 and 16:40, cards were charged but no orders were created. The graphs are green now; 212 customers still have money held for an order that does not exist, and no confirmation email to tell them what happened.
- Between 16:00 and 16:40, cards were charged but no orders were created. The graphs are green now; 212 customers still have money held for an order that does not exist, and no confirmation email to tell them what happened.
- Some of those customers tried again and were charged twice. Support learns this on Monday from a queue of angry emails, with no list of affected orders and no agreed answer to give.
- Finance finds the mismatch between payments and orders in the weekly reconciliation and opens its own investigation, because nobody told them the incident touched money.
- The summary says "a null check was missing in the order service". It is accurate and useless to everyone outside engineering, who wanted to know whether customers were harmed and what was done about it.
What is really going on
- An incident has two timelines. The technical one ends when the system is healthy. The customer one starts when the first person is harmed and ends when the last affected person has been told what happened and made whole. The second one is usually longer, and it is the one people remember.
- The harm is specific and countable: which customers, which orders, which window, what they lost — money held, a double charge, a missed delivery date, a promo that expired while they retried. If you cannot produce the list, you cannot make anyone whole, and "we are sorry for any inconvenience" is what you send instead.
- What you owe is a product decision, not a support script. Refund the duplicates automatically or wait for complaints? Honour the order at the old price? Send a voucher? Those choices have costs, precedents and legal edges, and they need someone with the data and someone with the authority in the same conversation.
- Silence is also a message. A customer who was double-charged and hears nothing concludes the shop does not know or does not care. The same customer who gets an email within a day saying "we charged you twice, the second charge is refunded, here is why" often trusts the shop more than before (Users and the Jobs They Hire Features For).
- Engineering is the only function that can produce the list. Support has the complaints, finance has the payments, but the join between "payment captured" and "order missing" lives in your tables. That is why the customer timeline is your job too.
Two timelines, one incident
Every customer-facing incident runs on two clocks. The system clock starts at the bad deploy and stops at the revert. The customer clock starts when the first person is harmed and stops when the last one has been told and made whole. On Friday the system clock read forty minutes. The customer clock was still running on Tuesday.
Most incident processes are built around the system clock — detection time, time to mitigate, time to resolve — because those are the numbers engineering can see. The customer clock is invisible to dashboards, which is exactly why someone has to own it deliberately (What Dashboards Hide).
The four questions after the revert
Once the system is stable, four questions replace "can we close this". They are asked in this order because each depends on the one before, and the first can only be answered from your tables.
- Who was affected? A list, not an estimate. Include the second-order cases: the retries, the abandoned carts that were charged, the promo codes that expired during the window.
- What did they lose? Money held, money taken twice, a delivery slot, a price, their time on hold with support. Be specific per group.
- What do we tell them? One message per group, in their words, sent by the people they already hear from.
- What do we owe them? A refund, the order they paid for, a voucher, nothing but an apology. Decided with the people who hold the budget, written down.
1-- Every capture in the window that has no order, plus captures that duplicate a successful one2SELECT p.customer_id,3 p.id AS payment_id,4 p.amount_cents,5 o.id AS order_id, -- null: paid, no order6 CASE WHEN o.id IS NULL THEN 'no_order'7 ELSE 'duplicate_charge' END AS harm8FROM payments p9LEFT JOIN orders o ON o.payment_id = p.id10WHERE p.captured_at BETWEEN '2026-09-18 16:00' AND '2026-09-18 16:45'11 AND (o.id IS NULL12 OR EXISTS (SELECT 1 FROM payments p213 WHERE p2.customer_id = p.customer_id14 AND p2.cart_id = p.cart_id15 AND p2.id <> p.id16 AND p2.captured_at BETWEEN '2026-09-18 16:00' AND '2026-09-18 17:30'));The window extends past the revert on purpose: customers who retried after 16:40 succeeded and were charged twice. A query that only looks for missing orders would never find them.
Telling the people who were harmed
The customer message is short, specific and about them. It says what happened to their order, what has already been done, and what — if anything — they need to do. It does not describe the architecture, and it does not apologise for "any inconvenience" when the inconvenience is known and countable.
Thirty-one customers retried checkout after the failure and were charged twice. The duplicate charges are being refunded. Support asks engineering what to send.
"Due to a technical issue in our order processing system, some customers may have experienced problems at checkout. We apologise for any inconvenience."
"On Friday between 4 and 5pm a problem on our side meant your first checkout attempt charged your card without creating an order. You then ordered again successfully, so you were charged twice. We refunded the first charge today; it should show on your statement within five working days. Your order is on its way as normal. You do not need to do anything."
The fix is one line of the response
In the write-up, the revert deserves one line, maybe two. The rest is the customer timeline: how many people, what they lost, what they were told and when, what they were given, and how long it took from revert to last customer made whole. That last number is the one to improve next time.
This is also where the product engineer earns the right to be in the room for the next "what do we owe" decision. The team that arrives with the list, a proposal and the cost of each option makes the decision easy. The team that arrives with "it is fixed now" gets told what to do.
SEV-2. Null pointer in order service after deploy 4812. Detected 16:22, reverted 16:40. Root cause: missing null check on promo field. Resolved.
212 customers charged without an order (16:00–16:40); 31 charged twice after retrying. Duplicates refunded Friday night; 181 orders created at the price shown, agreed with finance Saturday. All customers emailed by Saturday 12:00. Cause: a deploy made order creation fail after payment; reverted 16:40. Customer clock: 20 hours.
Everyone outside engineering can act on the second one — support knows what customers were told, finance knows what it cost, the PM knows how long harm lasted. The first only tells engineers what to grep for.
How to do it
Most important first.
- Declare the customer question at the start, not the end: the first message in the incident channel after "what is broken" is "who is affected and how would we find them?". Start the query while the fix is still being written.
- Produce the affected list as an artefact: order ids, customer ids, amounts, what happened to each. Hand it to support and finance with one line saying what it contains and what it does not.
- Decide what is owed in one short conversation with the PM, support lead and finance — with the list on the screen. Write the decision down with who made it (Decision Records).
- Send the customer message before the internal write-up. Plain words, what happened to them, what you have done, what they need to do (ideally nothing).
- Close the incident only when both timelines are closed. "Fixed" and "resolved" are different states; name them differently in the tracker.
How to explain the decision
The sentences, the order, and what to lead with — for someone who did not make the call.
- Lead with the people, not the cause: "212 customers were charged between 16:00 and 16:40 without getting an order. 31 of them were charged twice. We have the full list."
- Then what has already been done: "The duplicate charges are being refunded now, automatically. The remaining 181 need a decision: create the order they paid for, or refund."
- Then the decision you need and from whom: "I would create the orders at the price they saw, because they wanted the goods and we took the money. That needs finance to agree the promo cost. Can we decide in the next hour?"
- The cause comes last and in one sentence: "A change on Friday made order creation fail silently after payment. It is reverted; the write-up with the change we will make is due Wednesday (Postmortems That Change Things)."
- "We cannot email customers without legal review." Then legal review goes on the incident checklist with a named contact and a turnaround. The message still has to go out; the question is whether it goes out in a day or a week.
- "Refunding automatically is risky, what if the list is wrong?" Then check the list against payments before acting, and say how you checked. A wrong list is a reason to verify it, not to wait for complaints.
- "This is scope creep for on-call." On-call fixes the system. Someone on the team that shipped the feature owns the customer side; it is part of shipping it (On-Call for Product Engineers).
What can go wrong
- Over-counting to be safe: emailing every customer who visited checkout in the window, including the ones who were fine, and generating the support wave you were trying to avoid.
- Under-counting because the query only looked at one failure path. The customers who retried and succeeded were double-charged but have a valid order, so they do not appear in "orders missing".
- Promising compensation in the incident channel before anyone with budget has agreed, then retracting it.
- Writing the customer email in engineering language — "a regression in our order pipeline" — so it reads as an excuse rather than an account.
- Treating every alert as a product event. A failed background job nobody saw is an operational incident; the customer timeline only opens when a customer was actually harmed.
- "Customer communication is support's job." Support sends it; engineering is the only team that can say who should receive it and what happened to them.
- "If the outage was short it is not worth a customer message." Duration is a system metric. Forty minutes of charging without creating orders is a small window and a large harm.
- "We should be transparent, so explain the technical cause." Customers want to know what happened to them and what you did about it. The cause, in their terms, is one clause at most.
Knowing whether it worked
- The support queue after an incident contains follow-ups to your email, not first reports. Customers are hearing it from you before they notice.
- Finance hears about money-touching incidents from engineering the same day, not from reconciliation a week later.
- For every customer-affecting incident in the last quarter, someone can produce the affected list and say what each customer received.
- The incident tracker has a "customers made whole" field, and it is filled in.
- At ten customers, the founder emails each one personally, and that is the right response. At ten thousand, the affected-list query and a templated but honest message need to exist before the incident, because writing them under pressure is where the mistakes happen.
- At 10x revenue, "what do we owe" acquires a legal and accounting dimension: consumer-protection rules on refunds, chargeback windows, tax on vouchers. The product decision stays the same; the number of people who must agree grows.
- With a dedicated incident-management function, the customer timeline gets an owner by default. The product engineer's job shrinks to producing the list quickly and correctly — which is still the part nobody else can do.
- It keeps the incident open longer. Engineers who just spent two hours fixing something now spend another two producing a list and joining a conversation about vouchers.
- Proactive contact tells some customers about a problem they had not noticed. Most appreciate it; a few complain who otherwise would not have.
- Making people whole costs money, and the precedent sticks. Once you have honoured a price after one incident, customers and colleagues will expect it next time.
Where this applies
Product advice is context-sensitive. These labels say what each claim is specific to, and where a different stage, team or product would differ.
- GENERALAnywhere a failure reaches a person — a shopper, a B2B admin, a colleague on an internal tool — the two-timeline view holds. What changes by product is who you talk to, not whether.
- PRODUCT-SPECIFICConsumer checkout: the harm is usually money and the remedy a refund or honoured order. B2B platforms: the harm is your customer's customers, the message goes through an account manager and the contract may define what is owed; regulated products may have a mandatory notification window that overrides your own judgement.
- TEAM-SPECIFICWith an incident-management or customer-operations team, engineering supplies the list and the facts while they own the message. On a small team without one, the engineer who shipped the change often writes the customer email too, and should get it reviewed by whoever speaks to customers.
Where the depth lives
This domain teaches the product-side judgement and hands the mechanism off.