Performance and Observability Anti-Patterns
Every one of these is a plausible move that a competent engineer makes under pressure, and every one shares a single property: no measurement before, or no measurement after. That is the tell, and it is the only thing they have in common.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Performance anti-patterns
These are not strawmen. Each one is a reasonable-sounding response to real pressure, usually proposed by someone experienced, often in a meeting where the alternative is admitting nobody knows where the time goes. They are worth naming because naming them is what makes them refusable.
The common structure: a technique that *sometimes* works is applied without establishing that its precondition holds. Adding an index works when the query is index-eligible and the scan is the cost; it works badly when the write amplification exceeds the read benefit, when the planner ignores it, or when the actual cost was lock waiting (Low CPU, High Latency: Lock Contention). The technique is not wrong. Applying it without checking the precondition is.
Notice how many of these appear during incidents specifically. Time pressure converts "I do not know where the time goes" into "let us try the thing that worked last time", and last time's constraint is rarely this time's (The Bottleneck Moves After Every Fix).
| Anti-pattern | Why it is tempting | What it actually does | The check it skipped |
|---|---|---|---|
| Optimize without measuring | The hot spot feels obvious | Effort lands on code that was not the constraint | A profile or trace attribution (Measure Before You Optimize) |
| Cache everything | Caching helped before | Invalidation bugs, staleness, memory cost, stampedes | Which requests miss, and what a miss actually costs (Cache Stampede: Everyone Misses at Once) |
| Raise thread/worker count | More workers means more throughput | More contention and context switching; moves the queue downstream | Whether the constraint is concurrency or a downstream resource (Twenty Workers, All Busy, Five Hundred Waiting) |
| Buy a bigger instance | It is fast and requires no analysis | Masks the issue at recurring cost; useless if the constraint is I/O or a dependency | Which resource is actually saturated (USE: Utilization, Saturation, Errors) |
| Tune to the benchmark | The benchmark improves reliably | Wins on a workload no user has (Microbenchmark or End-to-End: Why p99 Did Not Move) | Whether the benchmark resembles production traffic |
| Ignore p99 | The average looks fine | The worst experiences stay invisible while the mean reassures | The distribution, not the mean (The Average Was Fine and Users Were Not) |
| No baseline | The change is obviously an improvement | Nothing can be validated or reverted with evidence | Measure before (Regression or Tuesday? Telling a Real Change from Noise) |
| Ignore queueing | Utilization looks acceptable | Latency explodes near the knee while utilization still reads "fine" | Wait time, not just utilization (Queueing: Why Systems Get Slow Before They Get Broken) |
| Log verbosely in hot paths | Logs are useful and feel free | Serialization CPU and I/O inside the critical path | Cost per request of the logging itself (The Log Bill and What It Is Buying) |
| Add indexes blindly | Indexes make queries fast | Write amplification, more storage, unused indexes the planner ignores | The plan, and whether scanning was the cost (An Index Scan Is Not Automatically Faster) |
| Scale workers when the DB is the constraint | The queue is growing, so add consumers | More concurrent load onto the saturated resource; often makes it worse | Which resource is saturated (The Backlog Arithmetic: Four Levers and a Drain Time) |
| Retry without backoff | Retries improve reliability | Amplifies load exactly when the dependency is failing | Whether the dependency is degraded or down (Retry Storms: The Load You Generated Yourself) |
Observability anti-patterns
The observability failures are subtler because their cost is deferred. An unbounded label does not break anything today; it breaks the metrics backend in four months, usually during an unrelated incident when you need it most (Cardinality: The Label That Took Down Monitoring). A dashboard with two hundred panels is not wrong, it is simply unusable at 3am, and the unusability only manifests when someone is under pressure.
The dashboard case deserves specific attention because it is so common. Dashboards accumulate: every incident adds a panel, nothing is ever removed, and after two years the service overview answers no question quickly. A dashboard should be built around the questions a responder asks in order — is it healthy, is traffic normal, are users slow, are errors rising, which dependency, is anything saturated — and everything not serving one of those belongs on a secondary page (Dashboards Built Around Questions).
The most expensive one is alerting on every threshold. It produces a rotation where most pages are noise, and a rotation where most pages are noise reliably produces one where a real page gets acknowledged and set aside (Alert Fatigue: The Page Nobody Reads). The failure is not the noisy alert; it is the trained response to all alerts that the noisy alert creates.
| Anti-pattern | Looks like | Deferred cost | Instead |
|---|---|---|---|
| Unbounded metric cardinality | A user_id or request_id label | Metrics backend degrades or falls over months later | Bounded labels; identifiers belong in traces and logs (Label Sets That Survive a Year) |
| Logging secrets | Logging the full request for debugging | Tokens and personal data in storage with wide read access | Redact at the logging boundary (What You Just Wrote Into a Log Half the Company Can Read) |
| Dashboards with no question | Two hundred panels, nothing removed | Nobody can triage from it under pressure | Build around the responder's question order (Dashboards Built Around Questions) |
| Tracing every payload | Full request/response bodies as span attributes | Storage cost, privacy exposure, slow trace queries | Attributes that identify, not attributes that duplicate (Trace, Span, Attribute, Status) |
| Alert on every threshold | CPU > 80% pages someone | Alert fatigue; real pages get ignored | Alert on user-visible symptoms and SLO burn (Burn-Rate Alerts: How Fast Is the Budget Going?) |
| No correlation ids | Each service logs independently | A cross-service failure cannot be reconstructed at all | Propagate a request id everywhere (Correlation IDs: Turning Lines Into a Story) |
| No deploy markers | Charts with no change annotations | "What changed?" takes twenty minutes instead of five seconds | Annotate from every change source ("What Changed?" — Deploy Markers and the Invisible Deploys) |
| No user-centric SLO | Resource dashboards only | The system is "healthy" while users are failing | Define an SLI from the user's perspective (SLIs: Measuring What the User Actually Feels) |
The tell
Every performance anti-pattern above is identifiable without domain knowledge, by a single question: what measurement established that this is the constraint, and what measurement will show that the change worked? If both answers exist, the change is engineering even if it turns out to be wrong. If either is missing, it is a guess even if it turns out to be right.
This is why "we tried it and it got faster" is weaker evidence than it feels. Without a baseline you cannot tell an improvement from normal variance, a traffic dip, or a cache that happened to be warm. Without knowing which resource was the constraint, you cannot tell whether your change or an unrelated coincidence produced the improvement (Correlation Is Not the Root Cause).
The organizational version is worth stating: teams do not adopt these anti-patterns because they are careless. They adopt them because measuring first is slower than acting, and acting is visible while measuring is not. The fix is cultural — make the baseline a required part of the change, so that "I do not know where the time goes" is an acceptable thing to say in a meeting where the alternative is a plausible guess.
1Ticket: "Checkout is slow — add Redis caching to the orders lookup"2 3· which requests are slow? not established4· where does the time go? not measured5· is the orders lookup on the path? assumed6· expected improvement? "should help a lot"7· how will we know it worked? "it should feel faster"8 9shipped. p99 unchanged (the time was in the payment10dependency). cache retained anyway. now there is an11invalidation bug waiting, and no one will connect it12back to this ticket.1Ticket: "Checkout p99 2.4s vs 500ms SLO"2 3· slow requests: /checkout only, all regions [p99 by route]4· time attribution: payment span 1.9s of 2.4s [trace p99]5· constraint: external dependency, not ours [CPU 22%]6· hypothesis: payment call is on critical path and7 blocks the response unnecessarily8· expected: p99 → ~0.5s if made async9· validation: p99 by route, 24h, vs last Tuesday10· guard: CI check on checkout critical-path11 span count; alert on payment p9912 13result: p99 2.4s → 0.31s. Redis never needed.The instinct in both cases was reasonable — checkout is slow, caching helps slow things. The difference is four measurements taken before the work, which cost twenty minutes and redirected a week of effort away from a component that was contributing 4% of the latency.
Key points
- Every performance anti-pattern is a technique applied without establishing that its precondition holds — the technique is not wrong, the missing check is.
- Observability anti-patterns defer their cost: unbounded cardinality, secret-bearing logs and purposeless dashboards all fail months later, usually mid-incident.
- Alert fatigue is not caused by any single noisy alert but by the trained response to all alerts that noisy alerts create.
- The universal tell: what measurement established the constraint, and what measurement will show the change worked?
- "We tried it and it got faster" without a baseline cannot distinguish improvement from variance, a traffic dip, or a warm cache.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Pressure → proposal: a plausible technique is proposed, usually one that solved a previous, different problem.
- 2Proposal → skipped check: the precondition (which resource is constrained, which requests are slow) is assumed rather than measured, because measuring is slower and less visible.
- 3Change → ambiguous result: the metric moves a little or not at all, and with no baseline the result cannot be distinguished from variance.
- 4Ambiguous result → retention: the change is kept because reverting requires evidence that was never collected, so its costs accrue permanently.
- 5Accumulation → system: over quarters, the system carries many such changes — caches nobody can invalidate, indexes nobody uses, workers nobody needs — each with a maintenance cost and no measured benefit.
- • "It got faster after we shipped it" — without a baseline and a controlled comparison, this is compatible with variance, a traffic dip, or an unrelated change.
- • "This worked last time" — last time's constraint is rarely this time's, and the technique that relieved it is only correct when the same resource binds (The Bottleneck Moves After Every Fix).
- • "More monitoring is better monitoring" — every metric, label, dashboard and alert has a cost, and past a point they collectively reduce the ability to answer questions.
- • "CPU > 80% needs an alert" — utilization without user impact is not actionable; alert on symptoms and burn rate instead (Alerts Worth Waking Someone For).
- • "We can clean up the dashboards later" — dashboards are only judged during incidents, and incidents are when cleanup is impossible.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • A baseline for the target metric over a representative window before any change — same weekday and hour, since traffic shape confounds shorter comparisons.
- • Attribution of where request time actually goes, from traces, before choosing which component to optimize ([[critical-path]]).
- • Utilization and saturation across all resources, to establish which one is the constraint rather than which one is familiar ([[use-method]]).
- • Metric series cardinality per metric name, tracked over time, so unbounded labels are caught while they are cheap to fix.
- • Alert-to-action ratio per alert rule: how often each alert fired and how often it led to a human doing something.
- • Require a baseline measurement and an attribution step before approving performance work — make "we do not know yet" an acceptable status.
- • State the expected improvement quantitatively before shipping, so an unexpected result is recognisable as a signal that the model is wrong.
- • Track metric cardinality as an operational metric and alert on growth, before the backend degrades ([[cardinality]]).
- • Prune dashboards and alerts on a schedule, judged by whether each panel answers a triage question and whether each alert has ever led to an action.
- • Revert optimizations that cannot demonstrate a measured benefit, rather than keeping them because reverting feels risky.
- • The change moves the target metric by roughly the predicted amount, compared against the same window on a comparable day.
- • The attribution shifts as predicted: time leaves the component you optimized and appears in the next constraint ([[bottleneck-migration]]).
- • Dashboard pruning is validated by timing a triage drill — how long does it take a responder to answer the six questions?
- • Alert pruning is validated by the alert-to-action ratio improving without any incident going undetected.
- • Requiring measurement before action genuinely slows the response, and during a severe incident a fast guess with a cheap rollback can be the right call.
- • Pruning dashboards and alerts risks removing the one panel that matters next quarter; keep them on a secondary page rather than deleting outright.
- • Cardinality limits reduce the dimensions available for debugging exactly when a novel problem needs a novel dimension.
- • A decision log is administrative overhead that only pays off across quarters and staff turnover, which makes it easy to abandon.
- • Add a CI benchmark or load-test assertion for the specific property you fixed, so the regression fails a build rather than a customer (Regression or Tuesday? Telling a Real Change from Noise).
- • Alert on cardinality growth, log volume growth and cost per request — the three costs that grow silently until they are expensive.
- • Keep a short performance decision log: what was measured, what was changed, what was expected, what happened. It is what stops the next team repeating a change that did not work.
- • Revisit dashboards after every incident: what did the responder need that was not there, and what did they scroll past?
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe ticket examples and their outcomes are constructed to show the reasoning gap. Whether any specific technique here is an anti-pattern depends entirely on whether its precondition was checked.
- WORKLOAD-SPECIFICSeveral entries — caching, indexes, worker counts — are excellent engineering in the right context. They appear here as anti-patterns only when applied without establishing that the constraint they address is the one binding.