Four Metric Types, Four Questions
A counter, a gauge and a histogram are not three ways to record a number — they are three different questions, decided at instrumentation time. Choosing wrong does not make the dashboard ugly; it makes the question permanently unanswerable, because the data you needed was never recorded.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Each type stores a different shape of truth
A counter stores a total that only increases. It answers "how often", because the interesting value is not the counter but its slope: rate(http_requests_total[5m]). A gauge stores whatever the value was at the instant of the scrape. It answers "how much right now" — queue depth, open connections, resident memory. A histogram stores counts per bucket, so it answers "what was the shape of this distribution", which is the only type that can produce a p99.
The fourth type, a summary, computes quantiles inside the process and exports the results. It answers "what was p99 on this one instance" — and stops there, because pre-computed quantiles from four instances cannot be combined into a fleet-wide p99. That single property decides most real choices between summaries and histograms.
The types are not interchangeable and the conversion only runs one way. A histogram can always give you a count (sum the buckets). A counter can never give you a distribution: the information was discarded at record time, and no amount of querying brings it back. This is why the choice is a one-way door — you find out you needed a histogram during the incident where you do not have one.
| Type | Question it answers | Typical metric | What it can never tell you |
|---|---|---|---|
| Counter | How often does this happen? | http_requests_total, errors_total | How long any individual one took |
| Gauge | How much is there right now? | queue_depth, db_connections_active | What happened between two scrapes |
| Histogram | What was the distribution? | http_request_duration_seconds | Which specific request was slow (that is a trace) |
| Summary | What was p99 on this instance? | client-computed quantiles | What p99 was across the fleet — quantiles do not average |
The classic wrong choice: a gauge where a counter belonged
Someone needs request volume on a dashboard, so they add a gauge called requests_per_minute and set it from a counter the application resets every minute. It looks correct. It survives review. It fails the first time it matters, because a gauge is only observed when the scrape happens — every request that arrived between two scrapes is invisible, and a traffic spike shorter than the scrape interval leaves no trace at all.
A counter has none of these problems, because it accumulates. Whatever happens between scrapes still shows up in the difference between two readings, and the monitoring backend derives the rate at query time over whatever window you ask for. You get per-second, per-minute and per-hour views from one series, and you can change your mind about the window a year later.
The general rule this case illustrates: record the raw accumulation, derive the interpretation at query time. Anything you compute before storing — a rate, an average, a percentage — is a decision you can never revisit, applied to data you no longer have.
1# application resets this every 60s2requests_per_minute.set(count_since_last_reset)3 4# 15s scrape interval, 60s reset window:5# scrape at t=0 → 0 (just reset)6# scrape at t=15 → 4007# scrape at t=30 → 9008# scrape at t=45 → 14009# scrape at t=60 → 0 (reset — the 1800 total is gone)10#11# A 5-second spike of 3000 requests lands between scrapes.12# It is not attenuated. It is not visible at all.1http_requests_total.inc() # monotonic, never reset by the app2 3# The backend derives whatever window you ask for:4# rate(http_requests_total[1m]) → per-second over 1 minute5# rate(http_requests_total[1h]) → per-second over 1 hour6# increase(http_requests_total[5m]) → count in the last 5 minutes7#8# The 5-second spike still raises the total, so it is still9# visible in every window that contains it.The gauge stored an interpretation (per-minute) computed before the data was needed; the counter stored the accumulation and left the interpretation to query time. Only one of them can answer a question nobody thought of in advance.
Histogram or summary: the aggregation question
Both a histogram and a summary can show you p99 on one machine. Only the histogram can show you p99 across a fleet, and that is almost always the number you want, because users do not care which of your twelve instances served them.
The reason is arithmetic, not implementation. A histogram exports bucket counts — "412 requests were faster than 100ms, 890 were faster than 250ms" — and counts add up across instances. Once summed, the quantile is estimated from the merged buckets. A summary exports "p99 was 840ms here", and there is no valid operation that combines four such numbers into a fleet p99. Averaging them is the mistake Percentiles: Which One, and How Many Users Is That? covers in detail; the result is a number that is neither p99 nor anything else.
The cost of the histogram is bucket cardinality: every bucket is a time series, so a ten-bucket histogram with four labels is ten times the storage of a counter with the same labels. That is the trade — histograms buy you aggregatable distributions and charge you per bucket. Choose bucket boundaries deliberately (see Histograms: A Distribution You Can Afford to Keep Forever) rather than accepting a default that has no relationship to your latency.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Counter `requests_total` rate | 1,240/s | Traffic is normal — this rules out a load spike, nothing more | normal |
| Gauge `inflight_requests` | 890 | Far above the usual ~40; requests are accumulating in the process | suspect |
| Histogram p50 | 48ms | The typical request is unaffected — the problem is not universal | normal |
| Histogram p99 | 4.2s | One percent of requests are 90x the median: a tail problem, not a throughput problem | smoking gun |
| Summary p99 (per instance) | 0.9s / 4.8s / 1.1s / 5.0s | Two instances are bad, two are fine — but these cannot be merged into one fleet number | suspect |
Key points
- Counter = how often, gauge = how much right now, histogram = what distribution, summary = per-instance quantiles that cannot be merged.
- Record the raw accumulation and derive rates and percentiles at query time; anything computed before storage is a decision you cannot revisit.
- A gauge is blind between scrapes — any event shorter than the scrape interval may leave no evidence whatsoever.
- Histogram buckets add across instances, so histograms give fleet-wide percentiles; summary quantiles do not, and averaging them produces a meaningless number.
- The type is chosen at instrumentation time and cannot be changed retroactively, so choose it against the question you will ask during an outage.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Question → dashboard: an incident asks "how many failed in this five-minute window"; the dashboard shows only a current-value gauge.
- 2Gauge → storage: the gauge recorded a point-in-time sample every 15s, so events between samples were never written down.
- 3Query → backend: no query can reconstruct a count from samples, because the intervening data does not exist in any form.
- 4Backend → responder: the responder falls back to counting log lines, which is slower, more expensive, and may itself be sampled (see The Log Bill and What It Is Buying).
- 5Responder → postmortem: the action item is "add a counter", which means the next incident of this shape is the first one you can actually measure.
- • "We have a metric for that" — having a metric named after the concept is not the same as having a metric that can answer the question.
- • "p99 is 4.8s on this instance, so fleet p99 is about 4.8s" — per-instance summary quantiles do not aggregate; the fleet number can be far lower or higher.
- • "The gauge shows normal, so nothing happened" — a gauge only reports the instants it was scraped, and outages are frequently shorter than a scrape interval.
- • "We can compute the histogram later from the logs" — only if the logs contain per-request durations, are not sampled, and are retained long enough. Usually at least one of those is false.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • For every dashboard panel, ask which stored series answers it: if the answer is "we would need to have recorded a histogram", you have found a gap before the incident does.
- • Check whether any duration metric is a gauge or a summary; those cannot produce a trustworthy fleet percentile.
- • Compare `rate(requests_total[5m])` against any per-minute gauge measuring the same thing — divergence tells you how much the gauge is missing.
- • Count histogram buckets times label combinations to know the real storage cost before adding a histogram (see [[cardinality]]).
- • Instrument durations as histograms with explicitly chosen buckets, so fleet percentiles are available without redeploying during an incident.
- • Replace derived gauges (`*_per_minute`, `*_percent`) with the underlying counters, and compute the derivation in the query.
- • Keep gauges for genuinely instantaneous quantities — depth, in-flight, resident memory — and pair each with a histogram or counter if the value between scrapes matters.
- • Reserve summaries for cases where only single-instance quantiles are meaningful, and document that they cannot be aggregated.
- • Take a real question from the last postmortem and answer it purely from stored metrics; if you cannot, the instrumentation is still incomplete.
- • Generate a load spike shorter than the scrape interval in a test environment and confirm it appears in the counter-derived rate.
- • Compare the histogram-derived fleet p99 against a trace-derived p99 over the same window; large disagreement usually means bad bucket boundaries.
- • Histograms cost one series per bucket per label combination — the fleet percentile is real, and so is the storage bill.
- • Deriving everything at query time makes dashboards slower and queries more complex than reading a pre-computed gauge.
- • Keeping both a counter and a histogram for the same operation duplicates cardinality; usually the histogram's implicit count series is enough.
- • Add a review checklist item: any new duration or size metric must be a histogram with justified buckets.
- • Alert on metric type changes in CI by diffing the exported metric families between builds.
- • Keep a short list of "questions this service must be able to answer" and re-test it after any instrumentation change.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe scrape intervals, latencies and counts here are invented to show the shape of each failure. Real scrape intervals range from 10s to several minutes and change the size of the blind window proportionally.
- ENVIRONMENT-SPECIFICWhether summaries can be aggregated, how buckets are stored, and what a "rate" function does depend on the metrics backend. The counter/gauge/histogram distinction is near-universal; the query syntax shown is Prometheus-style.