Operationsmetricsobservabilitypercentilescardinalityslo

API Metrics: Rate, Errors, Duration, Sizes

Four signals per endpoint — request rate, error rate by class, duration percentiles, payload sizes — labeled by route template, method and status class. The craft is in the labels: one high-cardinality label like user_id can melt the metrics system that was supposed to watch everything else.

Follow the failure

Frame the contract

API design starts with a consumer, a design question and a guarantee — never with a URL.

Design question
Can you answer "is this endpoint healthy, for whom, and compared to what we promised?" from metrics alone — without grepping logs?
Consumers
The on-call engineer deciding in ninety seconds whether the page is real; the team reviewing SLO compliance against what the contract promised; the API owner deciding which endpoint's latency budget gets next sprint's attention.
The promise
Per-endpoint metrics that make health, error class, latency distribution and payload trends queryable in seconds, at a cardinality the metrics system can survive for years.
Requirement→Consumers→Resource Model→Style→Contract→Validation→Authorization→Errors→Idempotency→Pagination→Versioning→Observability→Evolution→Trade-offs

The four signals, per endpoint

For an API the useful minimum is RED plus sizes: Rate (requests/sec), Errors (rate by status class, at least 4xx vs 5xx — they mean opposite things: 5xx is your bug, a 4xx surge is usually a consumer's), Duration (latency as percentiles, never averages), and Sizes (request and response payload percentiles, the early-warning system for Payload Size: 20KB, 200KB, 5MB drift). Each is labeled by route *template*, method, and status class — enough dimensions to isolate "POST /payments is slow" without enough to explode.

Percentiles are non-negotiable because latency distributions are violently skewed: an endpoint with a 30ms average can hide a 4-second p99, and the p99 is where your biggest customer lives. Track p50, p95, p99 per endpoint; and remember percentiles do not average across instances or windows — aggregate histograms, not precomputed percentiles, or your dashboard is fiction. Averages answer "how much capacity"; percentiles answer "what did users experience"; the contract's promises (see Consistency as a Contract Clause, SLOs) are made in percentiles.

Sizes are the habitually-skipped fourth signal, and the cheapest regression detector an API can have: p99 response size creeping up release-over-release is a contract getting fatter, visible weeks before a consumer complains — and a request-size histogram is how you learn what real callers send before you tighten a limit (see Large Requests and Documented Limits).

The per-endpoint minimum, as label sets
http_requests_total          {route="/projects/{id}/tasks", method="GET", status_class="2xx"}
http_request_duration_seconds{route="/projects/{id}/tasks", method="GET"}   # histogram → p50/p95/p99
http_response_bytes          {route="/projects/{id}/tasks", method="GET"}   # histogram
http_request_bytes           {route="/projects/{id}/tasks", method="POST"}  # histogram

# route is the TEMPLATE, never the raw path
# status_class (2xx/4xx/5xx) — full code only where clients branch (429, 409)

Cardinality is the budget you spend with labels

A time-series system stores one series per unique label combination, and series are what you pay for — in memory, storage, and query time. The arithmetic is brutal: 50 routes × 5 methods × 3 status classes × 20 instances is 15,000 series, comfortable anywhere. Add one label with unbounded values — user_id on a service with 100,000 users — and you have asked for 1.5 *billion* potential series. The metrics system does not gracefully degrade; it OOMs, throttles, or silently drops data, and the monitoring for every other service on it goes down with yours.

The classic cardinality bombs are all tempting: raw URL paths (every /projects/8231/tasks becomes its own series — always label by the route template), user or tenant ids ("we want per-customer dashboards"), API keys, request ids (never — that is what logs are for), and unbounded error strings. The rule: a label's value set must be small, bounded and known in advance. Per-customer questions are real, but they belong in logs and traces (which store events, not series) or in a deliberate top-N/tiered design — e.g. a tier label with five values, or exact per-tenant metrics for your ten largest contracts only.

One label from melting the monitoring
1http_requests_total{
2  path="/projects/8231/tasks",   # unbounded: one series per id
3  user="u_88412",                # 100k users → 100k multiplier
4  error="timeout after 3000ms…", # free text → infinite
5  status="500"
6}
7# potential series: routes × users × error strings × …
8# outcome: TSDB OOM; every team's dashboards go dark
Bounded labels; events go to logs
1http_requests_total{
2  route="/projects/{id}/tasks",  # ~50 templates
3  method="GET",                  # ≤ 7
4  status_class="5xx"             # 3 (+ named codes: 429, 409)
5}
6# ~50 × 7 × 5 × instances ≈ 10⁴ series: survivable for years
7# per-user, per-request detail → logs, joined on request_id

Both record the same traffic. The left one answers slightly richer questions for a week, then takes the monitoring platform down. Cardinality is a hard budget: spend it on dimensions with small, fixed value sets, and route unbounded detail to logs where events are cheap.

Metrics as the contract's scoreboard

API metrics earn their keep when they measure the *promises*. If the contract says p99 < 300ms and 99.95% availability, the SLO dashboard must compute exactly those numbers per endpoint, and alerting should fire on burn rate against them — not on raw CPU or a global error count that mixes your checkout with your health checks. Contract-specific clauses deserve contract-specific metrics: 429 rates tell you whether The Rate-Limit Contract limits are sized right; Idempotency-Replayed counts show how often retries actually happen; deprecated-endpoint request counts are the burn-down chart for Deprecation as a Process, Not a Label.

Two boundaries keep the system honest. Metrics tell you *that* and *how much*, never *which one* or *why* — the moment you need a specific failing request, you pivot to logs via Request IDs: The Contract's Correlation Clause; resist the urge to make metrics carry event-level detail, because that urge is exactly the cardinality bomb. And measure at the layer consumers experience: instrumenting only inside the service misses queueing at the gateway and TLS at the edge — the The Gateway as Policy Boundary layer sees what consumers feel, service-side metrics explain it (a persistent gap between the two *is* a finding).

Which question, which telemetry
QuestionMetricsLogsTraces
Is /payments erroring more than promised?Yes — error rate vs SLO, by routeNo — too slow to aggregate at query timeNo — sampled
Which request failed for customer X at 14:47?No — and adding user labels to try is the bombYes — filter on request_id / principalIf sampled in
Why is p99 slow — which hop?Locates the endpoint and the whenPer-hop timing if loggedYes — this is the trace's job
Is response size drifting release-over-release?Yes — size histograms per routePossible but expensiveNo

Key points

  • Instrument RED + sizes per endpoint: rate, errors by class, duration percentiles, payload size histograms.
  • Label by route template, method and status class — never raw paths, user ids, keys or free text.
  • Cardinality is multiplicative: one unbounded label turns 10⁴ series into 10⁹ and takes the monitoring platform down for everyone.
  • Percentiles over averages, aggregated from histograms — a 30ms average happily hides a 4s p99.
  • 4xx and 5xx are different stories: one is usually the consumer's bug, the other is always yours; alert on them differently.
  • Metrics say *that* and *how much*; logs (joined on Request IDs: The Contract's Correlation Clause) say *which one*; traces say *where inside*. Do not make one carry another's job.

Follow the failure

How the contract fails or gets misused, hop by hop — and what it costs when it completes.

  1. 1
    Team → dashboards: instruments a global request count and average latency; everything is green.
  2. 2
    Consumers → API: the p99 on one endpoint quietly triples; the average moves 2ms; no alert fires.
  3. 3
    Team → metrics: reacting to a big customer's complaint, adds user_id and raw path labels to "see everything".
  4. 4
    TSDB → org: series count goes vertical; the metrics platform OOMs and every team's alerting goes dark during the debugging of an unrelated incident.
  5. 5
    Team → logs: strips the labels back out, and the per-customer question is finally answered where it always lived — in logs, joined on request id.
What breaks
  • Silent SLO violations: averages and global counters stay green while specific endpoints and specific percentiles break promises.
  • A cardinality explosion is a monitoring outage — the blast radius is every service sharing the metrics platform, at exactly the moment someone was watching a graph.
  • On-call decisions degrade: without per-endpoint, per-class signals, every page starts with twenty minutes of orientation instead of ninety seconds of triage.

Design, observe, evolve

A contract decision is incomplete until you know how you would notice it failing and how it changes later.

Design the contract
  • • Standardize the four signals and their label set (route template, method, status class) in shared middleware so every service emits identically shaped metrics.
  • • Enforce a label-cardinality policy mechanically: allowlisted label keys, bounded value sets, CI or ingestion-time rejection for raw paths and ids.
  • • Define SLOs per endpoint in the same terms as the contract's promises, and alert on burn rate against them rather than raw thresholds.
  • • Emit histograms, not precomputed percentiles, so aggregation across instances stays mathematically valid.
Observe in production
  • • Watch the watcher: series count and ingestion rate per service, with alerts on growth spikes — a cardinality bomb announces itself in minutes.
  • • The gateway-vs-service duration gap per route: a widening gap means queueing or edge latency that service-side metrics cannot see.
  • • p99/p50 ratios for duration and size: distribution skew is the earliest signal that one class of caller is having a much worse day than the median.
Evolve without breaking
  • • New metrics and new bounded labels add compatibly; renaming or re-labeling breaks every dashboard and alert downstream — treat the metric schema as an internal API with its own [[backward-compatibility]] discipline.
  • • When per-tenant visibility becomes a product requirement, evolve deliberately: a bounded `tier` label, exact series for the top N contracts, or a separate analytics pipeline — never `tenant_id` on the hot path.
  • • Deprecation metrics (requests per deprecated route per consumer) turn [[consumer-driven-evolution]] from guesswork into a burn-down chart.
What it costs
  • • Bounded labels genuinely answer fewer questions — the long tail of per-customer, per-request curiosity is deliberately pushed to logs, which are slower to aggregate.
  • • Histograms cost more than counters (buckets × series), and good bucket boundaries require knowing your latency range in advance; bad buckets quantize away the truth.
  • • Per-endpoint SLOs and burn-rate alerting take real setup and tuning; the alternative — a global error threshold — is one line and wrong.

Misconceptions

Claim
“More labels means more visibility — label everything you might want to query.”
Reality
Series cost is the product of label value counts. One unbounded label multiplies everything else by infinity; the question you wanted answered per-user belongs in logs, which store events, not series.
Claim
“Average latency plus a global error rate is enough to know the API is healthy.”
Reality
Averages hide tails and global counters hide endpoints. "Healthy" is a per-endpoint, per-percentile claim, because that is how the contract's promises are written and how individual consumers experience the API.
Claim
“You can average percentiles across instances to get the fleet p99.”
Reality
Percentiles are not linear; the average of ten per-instance p99s is not the fleet p99 and can be wildly off. Aggregate the histograms, then compute the percentile.
← Request IDs: The Contract's Correlation ClauseAPI Logging Without Leaking →