Measuring What Matters

Picking one metric that moves when the product improves, instrumenting before building, reading an experiment honestly, guardrails against winning the wrong way, what a dashboard cannot tell you, and where qualitative signals beat numbers.

Picking a Metric That Moves When the Product Gets Better

One number, close to the job the user came to do, that goes up when the product genuinely improves and is hard to push up any other way — written down precisely enough that two people compute the same value.

Q · Which single number would go up if this part of the product genuinely got better, and would be hard to push up for any other reason?
Instrumentation First

Decide how you will know before you build, and ship the events a cycle ahead of the feature — because "measure it later" has no baseline, and a number with no before is a number with no meaning.

Q · If this ships next month and someone asks whether it worked, what will you compare it against?
Reading an Experiment Honestly

Sample size decided in advance, a duration that covers the weekly cycle, a stop rule nobody changes after peeking, novelty accounted for — and "+3%" reported with its range and its conditions, not as a fact about the future.

Q · The dashboard says the new checkout is up 3% — what would you need to know before you believe it?
Guardrail Metrics

The numbers that must not get worse while you move the one that should — chosen before the change, with thresholds agreed in advance, so that "we won" cannot mean "we won by breaking something nobody was watching".

Q · If your metric goes up next week, what else might have gone down to make it happen?
What Dashboards Hide

A dashboard shows what someone thought to measure when they built it. The support tickets, session recordings, error logs and sales notes show what nobody thought to measure — which is usually where the next problem is.

Q · If every line on the dashboard is green, what would have to be true for users to be having a bad week anyway?
Qualitative Signals

The chart tells you what happened; five conversations tell you why. How to find the people, ask about the last time instead of the hypothetical, keep your solution out of the room, and turn what you hear into something the team can act on.

Q · The chart shows people leaving at the shipping step — how would you find out why without asking them a question that already contains your answer?