Metrics
Counters, gauges and histograms as different questions. Why the average hides the outage, how to read p50 against p99, and why a user id in a metric label can take down the monitoring system.
Every lesson below starts from an observable symptom and ends with the measurement that proves the fix worked. Numbers carry a label saying whether they were measured, estimated, simulated or invented to show a shape.
A counter, a gauge and a histogram are not three ways to record a number — they are three different questions, decided at instrumentation time. Choosing wrong does not make the dashboard ugly; it makes the question permanently unanswerable, because the data you needed was never recorded.
A counter only goes up, which makes the raw value almost useless and its slope almost everything. The two things that go wrong: graphing the total instead of the rate, and mishandling the reset that happens every time the process restarts.
A gauge reports whatever the value was at the instant of the scrape. That is exactly right for queue depth and resident memory, and exactly wrong for anything that spikes — because a spike shorter than the scrape interval can leave no evidence that it ever happened.
A histogram stores counts per bucket instead of individual observations, which is what makes fleet-wide percentiles possible at all. The two decisions that determine whether it is useful: where you put the bucket boundaries, and whether you understand that every percentile it reports is an interpolation.
Five requests at 50, 55, 52, 48 and 3000ms have a mean of 641ms — a number no single request experienced. The mean is the wrong summary for latency because one extreme value drags it away from everything, and it cannot distinguish "everyone is slightly slow" from "one user in a hundred is unusable".
p50 describes the typical user, p99 describes the worst-served one percent, and the distance between them describes the system. The two errors that matter: reading a percentile without knowing the traffic volume behind it, and assuming percentiles compose across a call chain. They do not.
Cardinality is the product of every label's distinct value count, and it multiplies. One `user_id` label turns a three-series metric into three million, and the first thing that breaks is the monitoring system you were relying on to tell you what broke.
A label set is a schema: bounded value sets, names that mean the same thing in every service, and a migration path for the day you need to change one. Get it wrong and you either cannot join across services or cannot afford the series you created.