The Log Bill and What It Is Buying
Log cost scales with traffic while its debugging value does not — the ten-thousandth identical success line teaches nothing. Sampling is how you keep the value and drop the volume, and the rule that makes it safe is simple: never sample what you would need during an incident.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Where the volume actually is
Log volume is almost always dominated by a small number of very common events, and those events are almost always the least informative ones. A successful request logged at info on a service handling 8,000 requests per second is 690 million lines a day, of which you will read perhaps a hundred. The rare events — errors, slow requests, unusual paths — are a rounding error in volume and carry nearly all the diagnostic value.
That asymmetry is what makes sampling work. Keeping every error and one in a hundred successes preserves essentially all the debugging capability at a fraction of the cost, because the population you sample away is the one where every member is interchangeable with every other.
Before sampling, though, look for pure waste: debug lines left on in production, per-iteration logging inside hot loops, health-check and readiness-probe requests logged like real traffic, and multi-line stack traces emitted for expected failures. In most systems these account for a large share of volume and none of the value, and deleting them is strictly better than sampling them.
event lines/day share queried? verdict --------------------------------------------------------------------- request_completed 690,000,000 82.6% rarely sample 1:100 health_check 86,400,000 10.3% never drop entirely cache_lookup 41,000,000 4.9% never drop (metric covers it) job_processed 14,000,000 1.7% rarely sample 1:20 request_failed 310,000 0.04% always KEEP ALL payment_failed 21,000 0.003% always KEEP ALL auth_denied 8,400 0.001% always KEEP ALL --------------------------------------------------------------------- total 835,739,400 after dropping health_check + cache_lookup and sampling the rest: total ~7,600,000 -99.1% every high-value event kept in full
Head, tail and error-preserving sampling
Head sampling decides at the start of a request, before the outcome is known: keep one in a hundred, drop the rest. It is trivial to implement and cheap, and its weakness is decisive — you cannot keep all errors, because you do not yet know which requests will fail. What it does guarantee is coherence: if you keep a request, you keep all of its lines, so the sampled requests are complete stories rather than fragments.
Tail sampling decides at the end, when the outcome is known: keep everything that failed or was slow, sample the boring successes. This is what you actually want, and it costs more, because every line must be buffered until the request completes and something must hold that buffer. It is the same trade-off Sampling Without Throwing Away the Evidence describes for traces, for the same reason.
The practical compromise most teams land on: log errors and warnings unconditionally, head-sample successes at a rate that fits the budget, and make the sampling decision per request rather than per line so the surviving records tell whole stories. Then propagate the decision downstream, so a sampled-in request is sampled in everywhere and you do not get one service's half of a conversation.
| Strategy | Decision point | Keeps all errors? | Cost | Best for |
|---|---|---|---|---|
| Head sampling | Request start, outcome unknown | No | Negligible | High-volume success paths where errors are logged unconditionally |
| Tail sampling | Request end, outcome known | Yes | Buffering every line until completion | When you need slow and failed requests kept in full |
| Level-based | Per line, by level | Yes if errors are exempt | Negligible | The simplest useful baseline — pairs with head sampling |
| Per-event rates | Per line, by event name | Yes, by exempting high-value events | A rate table to maintain | Services with one or two dominant noisy events |
| Adaptive | Dynamic, driven by current volume | Usually, via exemptions | Complexity, and a rate that varies over time | Spiky traffic where a fixed rate over- or under-shoots |
What must never be sampled
Some records exist for reasons other than debugging, and sampling them is not a cost decision but a compliance or correctness failure. Security-relevant events — authentication decisions, authorization denials, privileged actions — are audit records, and a sampled audit trail is not an audit trail (see the audit-log material in Security Engineering). The same applies to anything a financial or regulatory process depends on.
Errors and warnings should be exempt by default. So should anything with a stable event name that you have ever used in an incident: if you queried it under pressure once, you will again, and its volume is almost certainly negligible compared to the success path.
The other lever is retention rather than sampling. Recent logs are queried constantly and old logs almost never, so tiered retention — full fidelity for a week or two, aggregated or archived thereafter — often saves more than sampling and loses less. And where the question is "how many", the honest answer is usually that it was never a log question at all: a counter answers it at a fraction of the cost and with better aggregation (see Four Metric Types, Four Questions).
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Ingest volume | 835M lines/day, 1.2 TB | The headline number; on its own it says nothing about what to cut | suspect |
| Share from one event | `request_completed` = 82.6% | One event dominates — sampling it is the single highest-leverage change | smoking gun |
| Health-check lines | 86M/day, never queried | Pure waste: drop rather than sample | smoking gun |
| Queries per event (30d) | `request_failed`: 1,400; `cache_lookup`: 0 | Query counts tell you what is actually earning its storage | normal |
| Error-level share | 0.04% of volume | Keeping every error in full costs essentially nothing — never sample these | normal |
Key points
- Volume is dominated by common, low-value events; diagnostic value is concentrated in rare ones, which is exactly what makes sampling viable.
- Delete pure waste first — debug in production, health checks, hot-loop logging — before sampling anything.
- Head sampling is cheap and cannot keep all errors; tail sampling keeps errors and costs buffering. Most teams combine head sampling with unconditional error logging.
- Sample per request, not per line, and propagate the decision downstream so surviving records are whole stories.
- Never sample audit-relevant events, and prefer a counter whenever the real question is "how many".
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Traffic → volume: request volume grows, and per-request logging multiplies it into hundreds of millions of lines a day.
- 2Volume → ingest: the platform charges on ingest and indexing, so cost tracks traffic exactly while debugging value does not.
- 3Ingest → quota: the budget is exceeded and someone shortens retention across the board to fit it.
- 4Retention → investigation: an incident needs logs from three weeks ago, and retention is now nine days.
- 5Investigation → outcome: the questions that mattered were about rare events whose total volume was under 0.1% — they could have been kept in full for essentially nothing.
- • "We need to cut logging in half" — cut the dominant low-value event by 99% instead and keep everything else intact.
- • "Sampling loses data" — it loses interchangeable copies of a common event, provided errors and audit records are exempt.
- • "Shorter retention is the cheapest lever" — it is the bluntest; it removes exactly the historical baseline incidents need.
- • "We can reconstruct counts from sampled logs" — only with the sampling rate applied correctly, and never for rare events. Use a counter (see Counters: The Slope Is the Signal).
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Break volume down by service, level and `event` name — one or two events usually account for most of it.
- • Cross-reference volume against how often each event has actually been queried in the last month.
- • Track cost per service and compare it against that service's compute cost to see when logging has become the larger bill.
- • Measure the age distribution of queries to size a tiered retention policy on evidence rather than guesswork.
- • Drop pure waste: production debug output, health-check and probe requests, hot-loop and per-item lines.
- • Exempt errors, warnings and audit events, then head-sample the dominant success events per request.
- • Propagate the sampling decision across services so a kept request is kept end to end.
- • Move "how many" questions to counters and tier retention by query age rather than truncating uniformly.
- • Confirm volume and cost dropped as predicted while the count of retained error and audit lines is unchanged.
- • Replay a past investigation against the new policy and check every line it needed still exists.
- • Verify sampled-in requests are complete across services rather than fragmented.
- • Head sampling means some failed requests have no logs at all, because the decision was made before the outcome was known.
- • Tail sampling requires buffering every line until the request completes — memory, complexity, and a failure mode of its own.
- • Aggressive sampling makes rare-event analysis statistically unreliable unless those events are explicitly exempt.
- • Tiered retention adds a second storage system and a slower query path for older data.
- • Alert on volume per service so a new noisy log call is caught within a day instead of at the next invoice.
- • Add a CI check flagging new log calls inside loops or on health-check paths.
- • Review the exemption list whenever a new high-value event name is introduced, so it is never sampled.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVELine counts, shares and the 99.1% reduction are constructed so the arithmetic is checkable. The distribution shape — one or two events dominating — is typical, but the exact proportions vary widely by service.
- ENVIRONMENT-SPECIFICWhether you are charged on ingest, indexed volume, retained bytes or queries changes which lever saves the most. Tail sampling support and tiered retention are platform features, not universally available.