Dashboards an Operator Can Act On
Built around the questions an incident asks, in the order it asks them — not around everything the system can emit.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What should be on the screen an operator opens at 3am, and what should not?
Dashboards are built by whoever instrumented the service, from what was easy to graph, and are then read by someone under time pressure who needs a specific answer.
Put every metric the service emits on one dashboard. Then whatever the question is, the answer is somewhere on the page.
Forty panels means the answer is on the page and cannot be found. Scanning is slower than querying, and an operator scans because they do not know where to look.
- Forty panels means the answer is on the page and cannot be found. Scanning is slower than querying, and an operator scans because they do not know where to look.
- Panels built from what was easy to collect encode the instrumenter's mental model, not the incident's question. "Requests by handler" is easy; "are checkouts failing" is what is being asked.
- Everything looks abnormal at 3am if you have never seen it at 3am. Without a baseline on the panel, a normal nightly dip reads as a symptom.
- Auto-scaled axes make a rise from 0.01% to 0.02% look identical to a rise from 1% to 40%, and both look alarming.
- A dashboard with no owner drifts: services get renamed, queries break, panels quietly return no data, and nobody notices until the incident.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A dashboard is an answering surface for a fixed set of questions. The design task is choosing the questions, in order, and then giving each exactly one panel.
- Incidents ask the same questions nearly every time, and they ask them in the same order: is it broken, how badly, for whom, since when, what changed, where in the path, and is the mitigation working.
- The top of the screen should answer "is it broken and how badly" without interpretation — that is where availability, latency and traffic for the primary user operations belong. Everything below it is for the questions that follow.
- A panel is useful when it supports a decision. "Error rate by endpoint" tells you where to look; "total request count" mostly tells you it is nighttime.
- Layout carries meaning under stress. Left to right and top to bottom is the order the operator reads, so it should match the order the questions arrive.
- The Observability & Performance domain owns panel and chart design in depth — percentiles, histograms, cardinality, what a heatmap is for. This lesson owns which questions the dashboard exists to answer (Dashboards Built Around Questions).
The questions, in the order an incident asks them
This is the design spec for the page. Build it top to bottom in this order and an operator can read down the screen as the incident progresses.
Each row names what the panel must show. If two panels answer the same question, one of them is decoration.
| # | Question | Panel that answers it | Common mistake |
|---|---|---|---|
| 1 | Is it broken? | Availability of the primary user operations | Showing per-handler request counts instead of success rate |
| 2 | How badly? | Error rate and latency percentiles against the SLO line | Averages, which hide the affected tail |
| 3 | For whom? | The same signal split by tenant, region or client version | A single aggregate line that cannot distinguish one tenant from all |
| 4 | Since when? | A window wide enough to show the onset, with a baseline overlay | A 15-minute range that starts after the problem did |
| 5 | What changed? | Deploys, flags and config annotated on the same axis | Change history in a different tool (Deploys on the Same Timeline as the Symptom) |
| 6 | Where in the path? | Dependency latency and error rate, per downstream | Only your own service's metrics |
| 7 | Is the mitigation working? | The panel from row 1, unchanged | Watching a proxy metric that recovers before users do |
Two dashboards for the same service
The difference is not effort — both took about the same amount of work. It is whether the builder started from what the system emits or from what an incident asks.
The left-hand version is not fictional; it is what you get by adding a panel each time someone wanted to see something.
row 1: CPU | memory | goroutines row 2: requests/sec by handler (18 lines) row 3: GC pauses | heap | fd count row 4: db queries/sec | pool size row 5: kafka lag by partition (32) row 6: p50 latency, all endpoints avg -> "is checkout broken?" is not on the page
row 1: checkout success rate <- SLO line
checkout p99 <- SLO line
DEPLOYS annotated on both
row 2: success rate by region | by tenant tier
row 3: dependency error rate + latency
(payments, inventory, auth)
row 4: queue backlog age | db pool saturation
link: runbook | traces | logs (prefiltered)
-> row 1 answers scope in one glanceThe left dashboard can answer almost any question given enough time, which is the resource an incident does not have. The right one answers the first four questions before the operator has finished reading, and it puts the machine metrics one row lower where they become diagnostic rather than distracting.
How dashboards go stale
Dashboards decay quietly. Unlike code, nothing fails when a dashboard becomes wrong; it just answers questions incorrectly on the day it is needed.
Each of these has been the reason an operator lost minutes during a real outage.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Metric renamed in a refactor | Panel shows a flat line at zero | Query still references the old name and returns no data | Alert on no-data for key panels; keep dashboards as code alongside the service |
| New region or tenant added | Incident affects it and the dashboard looks healthy | Panel filters to a hardcoded list | Use template variables driven by the label set rather than a fixed list |
| Service split into two | Half the traffic vanishes from the graph | The panel follows the old service name | Review dashboards as part of the change, like any other consumer |
| Traffic grows | Fixed y-axis clips the interesting range | Axis bounds set once, at an earlier scale | Prefer relative panels — error ratio, latency against the SLO — over absolute counts |
| Cardinality reduced to cut cost | "Which tenant?" can no longer be answered | The label the scope question depended on was dropped (Cardinality: The Label That Took Down Monitoring) | Treat scope-answering labels as load-bearing and cut elsewhere |
| Nobody owns the dashboard | Everyone builds a private copy | The shared one is untrusted, so it is bypassed | Assign ownership with the service (The Ownership Record) |
How to do it properly
Most important first.
- Write the questions first, in incident order, then build one panel per question. If a panel does not answer a written question, it belongs on a different dashboard or nowhere.
- Put user-visible symptoms at the top: availability and latency of the operations users perform, not internal handler counts.
- Overlay deploys and config changes on the same time axis as the symptom graphs. It is the highest-signal element on the page (Deploys on the Same Timeline as the Symptom).
- Show a baseline — last week at the same hour, or a comparison line — so an operator who has never seen this service at 3am can tell normal from abnormal.
- Split by the dimension that matters for scope: tenant, region, client version, endpoint. "All users or some users" is the question a dashboard most often fails to answer.
- Keep one dashboard per service that the alert links to directly, and let deep exploration happen elsewhere (Runbooks).
- Give it an owner and review it after each incident: which panel was used, which was missing, which was misleading.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing directly — a bad dashboard does not cause outages, it lengthens them, and it does so on every service that shares the template.
What can go wrong
- Panels that break silently when a metric name or label changes, showing an empty graph that reads as zero.
- Aggregation across tenants or regions hiding a scoped failure inside a healthy average.
- Averages instead of percentiles, which hide exactly the tail an incident lives in (Percentiles: Which One, and How Many Users Is That?).
- Time ranges too short to show that the condition started an hour before the alert fired.
- A dashboard so expensive to render that opening it during an incident adds load to the monitoring system.
- Panels showing the health of the monitoring pipeline's own inputs, which go blank precisely when they are needed.
- "Dashboards are how we monitor." Dashboards are how we investigate. Nobody is watching a screen at 4am — detection is the alert's job (An Alert Should Demand Action).
- "More panels means more visibility." Past a screen's worth it means slower answers.
- "The dashboard is green so we are fine." Green means the questions on the dashboard have good answers, which is a much smaller claim.
- "Every team should build their own." Every team should own theirs; starting from a shared template is what makes a stranger able to read it.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- An engineer who does not own the service can answer "what is broken, how bad, since when, what changed" from the top of the screen in under a minute.
- Alerts link directly to this dashboard, and it is the screen actually opened during the last incident.
- Every panel has a question written next to it or in its title.
- Dashboards defined as code roll back like any other config, which is worth the setup precisely because dashboards are edited during incidents by people under pressure.
- Keep the previous version reachable. A dashboard edited mid-incident is a change made without review, and reverting it should not require reconstruction from memory.
- Generate the standard per-service dashboard from a template so every service has the same top row and new services start with one (Service Templates).
- Automate deploy and change annotations onto the time axis from the delivery pipeline rather than relying on anyone to add them.
- Automate detection of dead panels — queries returning no data for an extended period — because a blank panel is indistinguishable from a healthy zero.
- Do not automate which questions matter. That is a judgement about the service and the users, and the generated template is only ever the starting point.
- A focused dashboard cannot answer unusual questions, and unusual incidents will need ad-hoc querying anyway. It is optimised for the common case on purpose.
- Templated dashboards are consistent and generic; a service with an unusual shape needs its own, and maintaining a bespoke one is ongoing work.
- Baselines and comparison overlays cost query time and can make the page slower to load at the worst moment.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe question-first structure holds for any monitoring product, because it follows from what an incident asks rather than from what a tool can render.
- TOOL-SPECIFICWhether annotations, baseline overlays and template variables exist at all depends on the platform. Where a tool lacks overlays, the substitute is a separate pinned change feed alongside the dashboard, which is worse but still answers the question.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — System Design — how a fleet-level view differs from a service-level one once there are more services than a person can hold in mind.