Gauges: Blind Between Scrapes
A gauge reports whatever the value was at the instant of the scrape. That is exactly right for queue depth and resident memory, and exactly wrong for anything that spikes — because a spike shorter than the scrape interval can leave no evidence that it ever happened.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
What a gauge is genuinely good at
A gauge is the right type whenever the quantity has no meaningful accumulation — when "how many are there right now" is the actual question. Queue depth, in-flight requests, open file descriptors, connection pool utilization, resident memory, replication lag: for each, the current value is what constrains the system, and summing it over time would be nonsense.
Gauges are also how saturation becomes visible. pool_connections_active / pool_size at 20/20 with pool_waiters at 800 is the signature of Connection Pool Saturation: Waiting in Front of an Idle Database, and neither number means anything as a total. Similarly, queue_depth rising steadily is the primary evidence in The Backlog Arithmetic: Four Levers and a Drain Time — though as that lesson argues, depth alone is a weaker signal than oldest-message age.
The trap is that a gauge feels like a complete record of the quantity, and it is not. It is a sequence of samples of a continuously varying value, with all the aliasing problems that implies.
| Gauge | What it constrains | Blind spot |
|---|---|---|
queue_depth | Backlog and therefore wait time | A burst that filled and drained between scrapes is invisible — see Depth Is Not an Emergency; Age Is for the more robust signal |
db_connections_active | Concurrency ceiling into the database | Brief full-pool episodes causing user-visible waits leave no gauge evidence |
process_resident_memory_bytes | Distance to the OOM killer | A short allocation spike that triggered a GC pause may never be sampled |
replication_lag_seconds | Staleness of reads from a replica | Lag that spiked and recovered within the interval is missed, though its effects on reads were real |
Aliasing: the spike that never happened
With a 30-second scrape interval, an event lasting 5 seconds has roughly a one-in-six chance of being sampled at all. Six such spikes an hour, and you may see one of them — or none. The dashboard is not lying about the samples it took; it is silent about the 25 out of every 30 seconds it did not observe.
This is the single most common reason a team concludes "the metrics show nothing" during a real, user-visible incident. The users experienced the 5 seconds. The gauge experienced the two instants either side of them.
There are three ways out, in increasing order of cost. Scrape faster, which multiplies storage across every series you collect. Export a max-since-last-scrape companion gauge, so a spike between samples still raises a number the scrape can see. Or record the underlying events as a counter or histogram, where nothing between scrapes is lost at all. The third is usually correct: if the transient matters, it is an event, and events belong in counters and histograms.
true value of pool_connections_active, one column ≈ 2.5s
20 | ████
16 | ████
12 | ████
8 | ▄▄ ▄▄▄ ▄▄ ████ ▄▄▄
4 | ██ ███ ▄▄ ██ ████ ▄▄ ███
0 +---------------------------------→ t
^ ^ ^
scrape scrape scrape
(val 6) (val 5) (val 7)
stored series: 6, 5, 7 → "pool is fine, peak 7 of 20"
what users hit: 20/20 for 5s → 800 requests queued behind the pool
with a max-since-last-scrape companion:
stored series: 9, 20, 8 → the 20 is visible, and it is the whole storyGauges that should have been something else
Any gauge whose name contains _per_second, _per_minute, _rate or _percent is a computed value that froze a decision at instrumentation time. The underlying quantity is almost always a counter (for rates) or two counters (for percentages), and deriving it in the query gives you every window instead of one — the argument made in full in Counters: The Slope Is the Signal.
Any gauge measuring a duration — last_request_duration_ms is the classic — is a histogram wearing a disguise. It tells you about one arbitrary request, chosen by the timing of the scrape rather than by anything meaningful. You cannot get a percentile out of it, and the percentile is what you wanted (see Percentiles: Which One, and How Many Users Is That?).
The reverse mistake also happens: treating an accumulating quantity as a gauge because it is convenient to set() it. If the value can be reconstructed by summing events, it wants to be a counter, and making it a gauge throws away everything between observations.
1gauge("requests_per_second").set(rps) # window frozen at instrumentation time2gauge("error_percent").set(errs / total * 100) # ratio computed too early to re-window3gauge("last_request_ms").set(duration) # one arbitrary request, chosen by scrape timing4 5# Questions none of these can answer:6# "how many requests in the 4-minute incident window?"7# "what did the slowest 1% experience?"8# "what was the error ratio over the last hour?"1counter("http_requests_total", labels=[route, status]).inc()2histogram("http_request_duration_seconds", buckets=[...]).observe(duration)3 4gauge("inflight_requests").set(n) # genuinely instantaneous5gauge("inflight_requests_max").set(max_since_last_scrape) # survives the blind window6 7# All three questions above are now answerable at query time,8# over any window, without redeploying anything.The gauges recorded interpretations of events; the counter and histogram recorded the events. Only the second survives a question nobody anticipated — and every incident asks at least one.
Key points
- Gauges are correct for quantities where "right now" is the question: depth, in-flight, memory, lag, pool utilization.
- A gauge is blind between scrapes — an event shorter than the scrape interval may leave no trace at all.
- A max-since-last-scrape companion gauge makes transient peaks visible without raising the scrape rate across the fleet.
- Names containing
_per_second,_percentor_durationusually mark a gauge that should have been a counter or a histogram. - "The metrics show nothing" during a real incident is most often a sampling artefact, not evidence that nothing happened.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Event → system: connection pool saturates for 5 seconds; 800 requests queue behind it and several hundred time out.
- 2System → gauge: the gauge holds the true value continuously, but nothing records it except at scrape instants.
- 3Scrape → storage: the two scrapes bracketing the event sample values of 6 and 5, both unremarkable.
- 4Storage → dashboard: the panel draws a flat, healthy line across the incident window.
- 5Dashboard → responder: the responder concludes the backend was fine and starts investigating the client, losing the first hour of the incident.
- • "The gauge was flat, so the resource was never saturated" — it was flat at the instants it was sampled, which is a much weaker claim.
- • "Memory looks stable" — a gauge sampled every 30s cannot rule out allocation spikes that triggered GC pauses (see Garbage Collection: Pause, Throughput, Footprint — Pick Two).
- • "Queue depth is low, so the queue is healthy" — depth says nothing about how long the oldest item has waited; see Depth Is Not an Emergency; Age Is.
- • "We need a faster scrape interval everywhere" — usually the cheaper fix is a max companion on the few gauges that matter.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Write down the scrape interval next to any gauge-based conclusion; it is the resolution limit of everything that gauge can tell you.
- • For saturation, read the utilization gauge together with a waiters or queue-depth gauge — full-with-no-waiters and full-with-800-waiters are different systems.
- • Add `*_max` companion gauges for pool utilization, queue depth and in-flight counts before concluding a transient did not occur.
- • Corroborate a suspicious "everything normal" gauge reading against counters, histogram tails, or traces from the same window.
- • Export max-since-last-scrape companions for the handful of gauges where transients cause user-visible harm.
- • Convert derived gauges (`_per_second`, `_percent`, `_duration`) into the underlying counters and histograms.
- • Raise the scrape interval only for a targeted subset of series, not fleet-wide, and price the storage first.
- • When a gauge shows normal during a confirmed incident, escalate to traces or logs from the same window instead of trusting the flat line.
- • Inject a saturation event shorter than the scrape interval in staging and confirm the max companion gauge records it while the plain gauge does not.
- • Replay a past incident window and check whether the new instrumentation would have shown the transient.
- • Compare gauge-derived peak utilization against trace-derived wait times over the same window; large disagreement means the gauge is under-sampling.
- • Max companions double the series count for those metrics and can look alarming — a peak of 20/20 is real but may last milliseconds.
- • Faster scraping multiplies storage and query cost across every series scraped from that target, not just the interesting one.
- • Converting derived gauges to counters moves work to query time, making some dashboards measurably slower to load.
- • Alert on the max companion rather than the instantaneous gauge for saturation-sensitive resources.
- • Record the scrape interval in the dashboard description so future readers know the resolution limit of every panel.
- • Review new gauges for names that imply a rate, ratio or duration and convert them before they reach production.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe 30-second scrape interval, the 5-second saturation event and the sampled values of 6/5/7 are invented to make the aliasing arithmetic concrete. Real intervals commonly range from 10s to 60s.
- ENVIRONMENT-SPECIFICWhether a max-since-last-scrape gauge is idiomatic depends on the client library and backend. Some stacks provide it natively; others require it to be maintained by hand.