What Dashboards Hide
A dashboard shows what someone thought to measure when they built it. The support tickets, session recordings, error logs and sales notes show what nobody thought to measure — which is usually where the next problem is.
The ask, the obvious response, and how it goes wrong
Every lesson starts where the work starts: someone asked for something, and the first response that comes to mind has a problem.
If every line on the dashboard is green, what would have to be true for users to be having a bad week anyway?
Monday review. The checkout dashboard is green: completion steady, order volume normal, latency fine. The PM says: "Checkout's healthy, let's spend this sprint on the order page."
The dashboard was built to show checkout health, all the numbers are normal, so checkout is healthy. Move on.
Support has had a steady trickle of "charged twice" tickets since a payment-provider change two weeks ago. Completion is fine — people did complete, twice. No tile on the dashboard counts duplicate charges.
- Support has had a steady trickle of "charged twice" tickets since a payment-provider change two weeks ago. Completion is fine — people did complete, twice. No tile on the dashboard counts duplicate charges.
- Customers on one mobile browser cannot tap "place order" because a cookie banner covers it. They leave and buy elsewhere. The dashboard shows a small dip in mobile completion inside normal weekly range, so nobody looks.
- Completion is steady because people who hit a confusing address form call support and order by phone. The dashboard counts those orders as completed checkouts. The user's experience is terrible and the number is fine.
- Nobody is lying and nothing is broken in the dashboard. It is answering the questions it was built to answer, accurately, while the new problem lives in a question nobody asked yet.
What is really going on
- A dashboard is a list of old hypotheses. Each tile exists because someone once thought "this might go wrong". It is structurally blind to problems nobody predicted — which are the ones most likely to be happening now.
- Aggregates average away small groups. A problem that affects one browser, one country or one payment method can be serious for those users and invisible in a site-wide rate.
- Metrics count what happened, not what it was like. A completed checkout after three failed attempts and a support call counts the same as a smooth one.
- Support tickets, session recordings, error trackers and sales call notes are unstructured on purpose: users describe the problem in their own words, which is exactly what makes them able to report things you did not think to count.
- Green dashboards create confidence that stops the looking. The better the dashboard, the less often anyone reads the tickets — and the longer an unmeasured problem survives.
A dashboard is a list of old hypotheses
Every tile on the checkout dashboard exists because someone once worried about that thing: completion, latency, payment errors. The tiles are accurate. They are also only the worries someone had when the dashboard was built. The problem this week is the one nobody worried about — and the places it shows up are the places with no tiles.
Green on what we measure
The phrase to retire is "checkout is healthy". The honest version is "checkout is green on the things we measure". The difference is small in a meeting and large in practice, because it keeps the question open long enough for someone to look at the tickets.
Monday review. The PM has just said checkout looks healthy and proposed moving on. You read the checkout tickets on Friday and saw a pattern.
"I think there might be some issues with payments, some customers have been complaining."
"Checkout is green on what we measure — but support has had twelve 'charged twice' tickets since the provider switch on the 3rd, and I found two in recordings where the pay button was double-tapped with no loading state. Our tiles don't count duplicate charges, so the dashboard can't see it. I'd like a day to confirm the cause; if it's the button, the fix is small and I'll add a duplicate-charge tile so we'd see the next one."
Aggregates average away the people with the problem
A site-wide rate is an average over every browser, country and payment method. A failure that hits one of them can be total for those users and barely visible in the total. Reading the dashboard by segment is the cheapest way to catch it.
Checkout completion 60.4%, within normal weekly range. No action.
Completion steady on every browser except one mobile browser, where it fell from its usual level to under half of it on Thursday — the day the new cookie banner shipped. That browser is a small share of traffic, so the total barely moved.
The segment view connects the drop to a specific release and a specific cause. The total hides both, because a large healthy majority outweighs a small broken group.
A weekly half-hour, and what to do with what you find
The habit that works is small and regular. It is not a research project; it is reading what users already told you.
- Tickets: filter for your area's tag, read the week's, and write down any pattern of three or more with the same shape.
- Recordings: watch five sessions that reached checkout and did not complete, or took far longer than usual. Not random ones.
- Errors: scan the error tracker and the payment provider's dashboard for anything new since the last release.
- Segments: look at the main tile split by browser, device and payment method once.
- Close the loop: any confirmed pattern becomes a ticket, a fix, and, if it could recur, a new tile — then check the tickets stop (The Support Loop).
How to do it
Most important first.
- Put reading the raw sources on a schedule, not on a trigger: a weekly half-hour with the support tickets tagged to your area, and a handful of recordings of sessions that did not complete (The Support Loop).
- Read by segment when you read the dashboard: browser, device, country, payment method, new versus returning. A flat total hides a broken slice.
- When a ticket pattern appears, turn it into a measure: a tag, an event, a tile. The dashboard should grow from what the raw sources teach you.
- Look at the error tracker and the payment provider's own dashboard alongside yours. Duplicate charges, declines and timeouts often show up there first.
- Treat "the dashboard is green" as the start of the question, not the answer: "green on what we measure — what did the tickets say?"
How to explain the decision
The sentences, the order, and what to lead with — for someone who did not make the call.
- Lead with what you saw, specifically: "Completion is fine, but support has had about a dozen 'charged twice' tickets since the provider change. None of our tiles count duplicate charges."
- Say why the dashboard missed it, without blaming it: "The dashboard was built before the provider change, so it doesn't have a question for this. It's answering what we asked; we didn't ask this."
- Size it honestly: "It's a small number of people, and for each of them it's the worst thing a store can do. I'd like a day to confirm the cause before we move on."
- Offer the permanent fix, not just the local one: "Once we've fixed it, I'll add a duplicate-charge count to the dashboard so the next one shows up there."
- "Reading tickets is support's job." Support reads them to help one customer. We read them to find patterns in the thing we built — nobody else can connect the ticket to the code change.
- "The dashboard is green, we don't have time to go looking for problems." The looking is thirty minutes a week. The problem we found last month through tickets took longer than that to explain to finance.
- "Session recordings are creepy." They can be. We mask all inputs, keep them for a short time, record only checkout, and follow whatever consent rules apply. If we can't do that, we don't record.
What can go wrong
- Reading recordings of random sessions, most of which are fine, and concluding everything is fine. Filter for the sessions that did not complete, or that took unusually long.
- Treating every ticket as a signal. A single angry ticket is a person; a pattern across a week is a signal. The skill is spotting the pattern without chasing each one.
- Adding every discovered problem as a new tile until the dashboard has sixty tiles and nobody reads any of them.
- Recording sessions without masking and consent, so the thing meant to protect users becomes a privacy problem of its own.
- "We need a better dashboard." A better dashboard is still a list of predicted problems. The fix is a habit of looking at the unpredicted ones.
- "Tickets are anecdotes, dashboards are data." Tickets are data about things you have not defined yet. They are where your next metric comes from.
- "If it mattered, it would show up in the numbers." Only if it is large, affects everyone, or you already count it. Serious problems are often small, local and uncounted.
Knowing whether it worked
- The team can name at least one problem found in tickets or recordings, not the dashboard, in the last quarter — and the dashboard now has a tile for it.
- Support brings patterns to the team before the team asks, because the last time they did, something changed.
- Segment-level views exist and get read: a problem on one browser is found in days, not at the next quarterly review.
- Tickets tagged to checkout go down after a fix, and someone checks that they did.
- At small scale one engineer can read every checkout ticket in a week. At 10x users that stops being possible; you need tagging, sampling and a support partner who knows what patterns to flag.
- At 10x team size, dashboards multiply and each one has a narrower view. The blind spots move to the boundaries between teams — the problem that starts in payments and shows up in the order page.
- At 10x revenue, a small segment is a lot of people. A problem affecting a sliver of sessions is still thousands of users a week, and "within normal range" stops being reassuring.
- Reading raw sources takes real time every week, and most weeks it finds nothing new. The payoff comes irregularly, which makes the habit easy to drop.
- Session recordings carry privacy cost and, in some jurisdictions, consent requirements. They need masking, retention limits and someone accountable for them.
- Qualitative sources are biased toward the users who complain. Quiet users who just leave never write a ticket, so tickets alone also hide things.
Where this applies
Product advice is context-sensitive. These labels say what each claim is specific to, and where a different stage, team or product would differ.
- GENERALAny dashboard is limited to what someone thought to measure; this is true of every product. It matters less for a new product with few users, where the team talks to most of them directly anyway.
- PRODUCT-SPECIFICConsumer products have support tickets and recordings as the main unstructured sources. B2B products have account managers, sales notes and direct customer channels, which are richer per customer and much less frequent.
- TEAM-SPECIFICWhere support tags tickets by product area, reading them is a filter away. Where tickets arrive untagged in a shared inbox, the first job is agreeing a tag with support — without it the habit will not survive.
Where the depth lives
This domain teaches the product-side judgement and hands the mechanism off.