Guardrail Metrics
The numbers that must not get worse while you move the one that should — chosen before the change, with thresholds agreed in advance, so that "we won" cannot mean "we won by breaking something nobody was watching".
The ask, the obvious response, and how it goes wrong
Every lesson starts where the work starts: someone asked for something, and the first response that comes to mind has a problem.
If your metric goes up next week, what else might have gone down to make it happen?
The PM wants an "add a matching item" upsell on the payment step to raise average order value. Finance is enthusiastic. The brief lists one metric: average order value.
Measure average order value, run the test, ship if it goes up. The other numbers are someone else's problem, and nothing in the change touches them.
Average order value rises. Three weeks later support notices more "I didn't mean to add this" contacts and returns on accessories. The upsell raised baskets by being easy to tap by accident.
- Average order value rises. Three weeks later support notices more "I didn't mean to add this" contacts and returns on accessories. The upsell raised baskets by being easy to tap by accident.
- Completion drops slightly because the payment step is now slower and busier — but nobody was watching completion in this test, so the loss in orders is never set against the gain in basket size.
- The team that owns returns sees its costs rise and has no idea why, because the cause is on a page they do not own and in a test they never heard of.
- When the damage is found, it is found informally, weeks later, and the argument is about whether the upsell caused it — an argument a guardrail agreed in advance would have settled on day three.
What is really going on
- Every product change moves more than one thing. The primary metric is the one you are trying to move; the others move too, and the ones you are not watching can only get worse without anyone noticing.
- Optimisation finds the cheapest path to the number, and the cheapest path often spends something unmeasured: user trust, support time, page speed, a neighbour team's costs.
- A guardrail is a metric you are not trying to move, with a threshold beyond which you stop regardless of how the primary looks. It turns "that seems bad" into "we agreed that would stop us".
- Guardrails come in three kinds: the user-harm ones (refunds, complaints, accidental actions), the business-health ones (orders, margin, the metric one level up), and the system ones (latency, error rate, payment failures). A change touches at least one of each more often than people expect.
Every number has a cheap way up
An upsell on the payment step raises average order value by making it easy to add one more item. "Easy to add" can mean "useful" or "easy to add by mistake". The primary cannot tell the difference. The guardrails can: accidental additions show up as removals, returns and support contacts.
Guardrails with thresholds, agreed before
A guardrail without a threshold is a chart. Agree the threshold with the team that owns the number, and put it in the experiment config next to the primary, so the readout computes it automatically and nobody has to remember.
1experiment: payment-step-upsell2primary:3 metric: average_order_value4 expect: increase5guardrails:6 - metric: items_removed_within_24h_rate7 stop_if: relative_increase > 10%8 owner: returns-team9 - metric: support_contacts_per_order # tag: added-item10 stop_if: relative_increase > 10%11 owner: support-lead12 - metric: checkout_completion13 stop_if: relative_decrease > 2%14 owner: checkout-team15 - metric: payment_step_p95_latency_ms16 stop_if: absolute_increase > 15017 owner: payments-platform18read_on: 2026-10-12The thresholds here are illustrative — the real ones come from a conversation with each owner about what harm would matter to them. The owner column is what makes the conversation happen.
When the primary wins and a guardrail loses
Sometimes the trade is worth it. The point of a guardrail is not to forbid trade-offs; it is to make them visible and make someone own them (Naming the Trade-off).
Readout day. Average order value is up clearly. Items removed within 24 hours rose past the threshold agreed with the returns team. The PM wants to ship.
"Order value is up, which was the goal. There's a small uptick in removals, but that's expected with more items in baskets."
"Order value is up. Removals within a day went up past the line we agreed with returns, and the recordings show people tapping the upsell while reaching for the pay button — it sits right above it. So some of the gain is accidental. I'd stop, move the upsell away from the pay button, and rerun for two weeks. If removals stay flat then, we ship with a clean result."
| Result | What it usually means | What to do |
|---|---|---|
| Primary up, guardrails flat | The change did what it was for without visible cost | Ship, with the usual post-launch check |
| Primary up, a user-harm guardrail up | The gain may be coming from mistakes or confusion | Stop; look at recordings and tickets before any redesign |
| Primary up, a system guardrail up | A real trade: more value, slower or less reliable | Fix the cost if cheap; otherwise write the trade down and get sign-off from the owner |
| Primary flat, guardrail worse | All cost, no gain | Stop and keep the old version |
How to do it
Most important first.
- For every primary, ask three questions: how could this go up by hurting the user, by hurting the business elsewhere, or by hurting the system? Each answer is a candidate guardrail.
- Keep the set small — typically three to five — and make each one measurable within the test's duration. A guardrail you can only read in six months is not a guardrail for this test.
- Agree the threshold before starting, with the people who own each number: "if refund rate on upsold items rises by more than a point, we stop". Numbers negotiated afterwards always end up generous.
- Include the metric one level up. For an upsell, orders per visitor or revenue per checkout session: the upsell should not raise basket size by losing baskets (Picking a Metric That Moves When the Product Gets Better).
- Tell neighbouring teams what you are testing and which of their numbers are guardrails; they will spot damage you cannot (Writing for Stakeholders).
How to explain the decision
The sentences, the order, and what to lead with — for someone who did not make the call.
- Lead with the way it could go wrong, not with the process: "An upsell on the payment step can raise basket size by being easy to tap by mistake. If that happens, we'll see it in returns and support before we see it in revenue."
- Name each guardrail and who owns it: "We're watching items removed within a day, support contacts per order, and completion. Support and the returns team have agreed the thresholds."
- Make the stop rule concrete: "If contacts per order go up by more than we agreed, we stop the test, whatever order value is doing."
- Say why this is good for the PM, not a hurdle: "It means if order value goes up, nobody can come back in a month and say it cost us something — we'll already have checked."
- "This is slowing down a simple test." Agreeing three thresholds takes a meeting. Discovering the upsell caused a spike in returns, after launch, takes a quarter to unwind and costs finance's trust.
- "Support contacts are too noisy to use." Then we use the specific tag — contacts mentioning an added item — which is quieter, and we agree the threshold with support before we start.
- "We'll catch problems after launch anyway." Some of them, weeks later, without a comparison group. During the test we can see exactly what the change did to each number.
What can go wrong
- Guardrails that are listed but not wired into the readout, so they are never actually checked.
- Guardrails with no threshold, which turn every degradation into a debate about whether it matters.
- Guardrails that lag: refund rate on a 30-day return window cannot be read in a three-week test. You need a faster proxy, like "removed within 24 hours" or support contacts.
- Over-guarding: fifteen guardrails, one of which will wobble by chance in every test, so nothing ever ships and the team learns to ignore them.
- "Guardrails are just secondary metrics." Secondary metrics are things you look at; guardrails are things that stop you. The threshold is what makes it a guardrail.
- "If nothing in the change touches it, it doesn't need guarding." The upsell does not touch returns code. It touches returns anyway, through people.
- "The primary going up proves nothing was harmed." It proves the primary went up. Harm lives in the numbers nobody was watching.
Knowing whether it worked
- Every experiment readout has a guardrail table, and at least one past test was stopped or changed because of it.
- Neighbouring teams — returns, support, payments — hear about checkout tests from the checkout team, before they see it in their own numbers.
- Post-launch surprises in other teams' metrics become rare, and when they happen, the question is "which guardrail did we miss" rather than "who broke this".
- Support tickets about the tested area stay flat through a launch, and someone looked.
- At 10x traffic, guardrails detect smaller harms, which is good and also noisy; thresholds need to be about harm that matters, not any change that is detectable.
- At 10x team size, some guardrails become shared: page latency, payment error rate and support contacts are guardrails for every checkout-adjacent team, and are best defined once and owned by someone.
- At 10x revenue, a small percentage change in refunds or payment failures is a lot of money; finance will want to own some guardrail thresholds directly, and should.
- Every guardrail is another chance for a false alarm. More guardrails means more tests stopped or delayed by chance wobbles.
- Agreeing thresholds with other teams takes time and meetings, before a test that might have been fine.
- Guardrails make some good changes impossible to ship: a change that genuinely trades a little latency for a lot of completion needs an explicit exception, and someone has to be willing to argue for it.
Where this applies
Product advice is context-sensitive. These labels say what each claim is specific to, and where a different stage, team or product would differ.
- GENERALAny optimisation can reach its target by spending something unmeasured; naming what must not get worse applies to every metric-driven change. It matters less for pure bug fixes, where the primary is "the error stops".
- PRODUCT-SPECIFICIn a store, user-harm guardrails are returns, accidental purchases and complaints. In a regulated product — payments, health — some guardrails are legal obligations with thresholds you do not get to negotiate at all.
- TEAM-SPECIFICWhere a central analytics or experimentation team exists, a standard guardrail set is often built into the tool. Without one, the engineer running the test has to choose them and ask the owning teams for thresholds.
Where the depth lives
This domain teaches the product-side judgement and hands the mechanism off.