Histograms: A Distribution You Can Afford to Keep Forever
A histogram stores counts per bucket instead of individual observations, which is what makes fleet-wide percentiles possible at all. The two decisions that determine whether it is useful: where you put the bucket boundaries, and whether you understand that every percentile it reports is an interpolation.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Buckets, and why they add up
A histogram does not store your 40 million request durations. It stores a small set of cumulative counters — "how many were ≤ 5ms, how many ≤ 10ms, how many ≤ 25ms" — and increments every bucket a given observation falls under. Storage is constant per bucket regardless of traffic, which is the whole point: a metric whose cost does not scale with the thing it measures.
The property that makes histograms indispensable is that counts add. Instance A's "≤ 250ms: 8,400" and instance B's "≤ 250ms: 7,900" merge into "≤ 250ms: 16,300" by simple addition, and the fleet-wide percentile is then estimated from the merged buckets. No other summarization has this property — which is why Four Metric Types, Four Questions treats summaries as a fundamentally different tool rather than a cheaper histogram.
The price is resolution. You do not know where inside a bucket an observation landed, so every quantile is an interpolation between two boundaries. If p99 falls in a bucket spanning 1s to 5s, the reported p99 is a guess inside a four-second range, and its accuracy depends entirely on whether someone chose boundaries that match your actual latency.
Bucket choice is the whole game
Default buckets are chosen by a library author who has never seen your service. If your median is 30ms and the default buckets are [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10] seconds, you have three buckets covering everything below 100ms — where 90% of your traffic lives — and eight buckets covering a range you almost never reach.
The rule that works: put boundaries where your decisions are. If the SLO says 95% of requests under 300ms (see SLOs: A Target, a Window, and a Reason), there must be a boundary at exactly 300ms, or the SLI is computed by interpolating across it and the compliance number is an estimate of an estimate. Add boundaries around the current p50, p90 and p99 so the interesting region has resolution, and keep coarse buckets in the tail where you only need to know "something was very slow".
Every boundary costs one time series per label combination, so bucket count multiplies with cardinality (see Cardinality: The Label That Took Down Monitoring). Ten buckets across four routes and three status classes is 120 series for one metric. That is affordable and deliberate; the same histogram with a customer_id label is not affordable at all.
| Aspect | Library defaults | Chosen for this service |
|---|---|---|
| Boundaries (ms) | 5, 10, 25, 50, 100, 250, 500, 1k, 2.5k, 5k, 10k | 10, 20, 30, 50, 80, 120, 200, 300, 500, 1k, 5k |
| Resolution at p50 (30ms) | One bucket spans 25–50ms — ±25ms of error | Boundary at exactly 30ms — p50 is nearly exact |
| SLO compliance at 300ms | Interpolated across the 250–500ms bucket | Read directly off the 300ms bucket — no interpolation |
| p99 (≈380ms) | Guessed inside a 250ms-wide bucket | Guessed inside an 200ms-wide bucket, one boundary away |
| Series per label combo | 11 | 11 — same cost, far better placed |
What the number actually is
When a dashboard says "p99 = 380ms", the honest expansion is: "we counted 37,100 observations at or below 250ms and 37,600 at or below 500ms; the 99th-percentile observation is somewhere in that gap, and assuming observations are spread evenly across the bucket, it lands near 380ms." The even-spread assumption is usually false — latency clusters — so the true value is often nearer one boundary than linear interpolation suggests.
This has a practical consequence people trip over constantly: a reported percentile can never exceed the highest finite bucket boundary in a meaningful way. If your top finite bucket is 10s and requests are taking 40s, everything past 10s lands in +Inf, and the p99.9 the dashboard reports is bounded by your instrumentation rather than by reality. The tell is a percentile that sits suspiciously close to a bucket boundary and refuses to move during an incident.
None of this makes histograms untrustworthy — it makes them approximate in a way you can reason about, which is exactly what you want from a metric that costs nothing per request. When you need the true value of one specific slow request, that is a trace, not a metric (see Metrics, Logs, Traces, Profiles).
total observations 38,000
99th percentile position 0.99 x 38,000 = 37,620th observation
cumulative counts:
le 250ms → 37,100 (below the target position)
le 500ms → 37,600 (still below — 37,600 < 37,620)
le 1000ms → 37,850 (crosses it)
so p99 lies in the 500–1000ms bucket
interpolate: 500 + (37,620 - 37,600) / (37,850 - 37,600) x 500
= 500 + (20 / 250) x 500
= 540ms
reported "p99 = 540ms" really means:
"between 500 and 1000ms, probably nearer the bottom,
assuming even spread inside the bucket (it is not)"Key points
- A histogram stores bucket counts, so its cost is constant per bucket regardless of how many requests it observes.
- Bucket counts add across instances, which is what makes fleet-wide percentiles possible; pre-computed quantiles do not add.
- Every reported percentile is an interpolation inside a bucket — its accuracy is entirely determined by boundary placement.
- Put a boundary exactly where a decision is made, especially at the SLO threshold, so compliance is read rather than estimated.
- Buckets multiply with label combinations, so bucket count is a cardinality decision as much as a resolution decision.
Progressive depth
Overview
A histogram counts how many observations fell into each bucket instead of storing every observation. That keeps cost flat as traffic grows, and it is the only common metric type from which a fleet-wide percentile can honestly be computed.
Practical
Choose boundaries where your decisions are: dense around the current p50 to p99, an exact boundary at every SLO threshold, and one high finite boundary above your worst realistic latency. Then read p50, p90, p99 and p99.9 together — a single percentile in isolation hides whether the problem is universal or confined to the tail.
Advanced
Every reported percentile is an interpolation between two boundaries under an even-spread assumption that latency distributions violate. The error bar is the bucket width, so a p99 inside a 250ms-wide bucket is a 250ms-wide claim. When percentiles stop responding during an incident, suspect that observations have piled into +Inf and the metric is now bounded by instrumentation rather than by the system.
Internals
Cumulative-bucket histograms export one monotonic counter per boundary, which is why merging across instances is plain addition and why resets follow the same rules as any counter (see Counters: The Slope Is the Signal). Sketch-based encodings such as DDSketch and HDR histograms trade this simplicity for relative-error guarantees that hold across the whole range, at the cost of a more complex merge operation and a backend that understands the encoding.
Percentile Explorer
Change an input and watch which number moves — and which one does not.
The tail is now heavy enough to drag the mean above p95 — the average has stopped describing any request that actually happened.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Requests → histogram: each duration increments every bucket whose boundary it falls under; the individual value is discarded.
- 2Histogram → storage: one time series per boundary per label combination, constant in traffic.
- 3Storage → query: the backend sums bucket counts across instances, then locates the target percentile position.
- 4Query → interpolation: the percentile falls between two boundaries, and a linear assumption produces the reported number.
- 5Interpolation → dashboard: a precise-looking "540ms" is displayed for a value known only to lie within a 500ms range.
- • "p99 is exactly 540ms" — it is an interpolated estimate whose error bar is the width of the containing bucket.
- • "p99.9 is 1.9s and stable, so the extreme tail is under control" — if the top finite bucket is near 2s, the metric physically cannot report worse.
- • "Averaging the four instances' p99 gives fleet p99" — quantiles are not linear; merge the buckets instead.
- • "Default buckets are a sensible starting point" — they are a starting point chosen without knowledge of your latency, and usually have almost no resolution where your traffic actually lives.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Read p50, p90, p99 and p99.9 together and note which bucket each falls into — a percentile inside a very wide bucket is a weak number.
- • Check that a boundary exists at every SLO threshold before trusting an SLI computed from the histogram.
- • Watch the `+Inf` bucket: growth there means observations are exceeding your highest finite boundary and the tail is unmeasured.
- • Multiply bucket count by label combinations to price the histogram before adding it (see [[cardinality]]).
- • Choose bucket boundaries from your observed distribution: dense around p50–p99, plus an exact boundary at every SLO threshold.
- • Add a high finite boundary well above your worst realistic latency so the tail is bounded by reality rather than by instrumentation.
- • Prefer histograms over summaries anywhere fleet aggregation is needed, and document the exception where a summary is deliberate.
- • Re-tune buckets when the distribution shifts materially — after an architecture change, boundaries chosen for the old system mislead.
- • Compare histogram-derived p99 against p99 computed from a sample of traces over the same window; agreement within a bucket width means boundaries are adequate.
- • Confirm the SLI reads directly off a boundary by checking the query does not interpolate across the threshold.
- • After re-bucketing, verify that p50 and p99 land in narrower buckets than before, and that series count changed as predicted.
- • More buckets means better resolution and linearly more time series — the cost is real and multiplies with every label.
- • Histograms give aggregatable approximations; when you need the exact duration of one request you still need traces.
- • Re-bucketing breaks historical comparability: old and new series are not directly comparable across the change.
- • Alert when the
+Infbucket rate rises — it means the measurable range no longer covers reality. - • Review bucket boundaries as part of any change that alters the latency profile (new dependency, new cache, new region).
- • Track series count per histogram in CI so a new label does not silently multiply storage.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEAll bucket counts, boundaries and derived percentiles here are invented to make the interpolation arithmetic checkable by hand. Real distributions are far less tidy.
- ENVIRONMENT-SPECIFICCumulative buckets and linear interpolation describe Prometheus-style histograms. Other systems use sparse or exponential-bucket encodings (DDSketch, HDR histograms, native histograms) with different accuracy guarantees and different failure modes.
Misconceptions
+Inf bucket, p99.9 is bounded by your highest finite boundary — the number stops responding to reality and starts reporting your instrumentation.