Learn Observability & Performance

Monitoring answers the questions you thought to ask when you built the dashboard. Observability is whether you can answer a question nobody anticipated — without shipping new code first. The test is not how many tools you run; it is what you can ask at 03:00.

The single most expensive habit in performance work is proposing a fix before taking a reading. This is the loop that replaces it: Problem → Measure → Locate → Understand → Change → Measure Again — and the six questions that turn "it is slow" into a specific reading at a specific layer.

The diagnostic chain — Symptom → Signal → Measurement → Hypothesis → Evidence → Root Cause → Change → Validation → Regression Check — and the discipline that makes it work: write the hypothesis down before you look, so you can be wrong out loud instead of quietly.

Four signals, four different questions. Metrics tell you something changed; traces tell you where the time went; logs tell you what exactly happened; profiles tell you what the CPU was doing. No single one explains an incident, and knowing which to reach for first is most of the speed.

Latency, traffic, errors and saturation. Four numbers that describe almost any request-serving system well enough to know whether it is healthy and, when it is not, which direction to look. The value is not the list — it is that the four are read together.

Three numbers per request-handling service: how many, how many failed, how long they took. RED is the fastest way to make every service in a fleet legible in the same shape — and it goes blind the moment work stops being request-shaped.

For every resource, ask three questions: how busy is it, is work waiting for it, and is it failing? The middle question is the one that matters and the one most dashboards omit — which is why "CPU is only 40%" keeps getting offered as evidence that CPU is fine.

Telemetry does not appear; it is emitted by code, batched by a client, shipped to a collector and stored by a backend — and every hop can drop data, add latency or cost money. Knowing the path is what lets you trust the dashboard, and notice when it lies.

A vendor-neutral way to describe traces, metrics and logs, propagate context across process boundaries, and ship the result anywhere. Worth understanding as a set of concepts — signals, context, semantic conventions, collector — rather than as a product to install.

A counter, a gauge and a histogram are not three ways to record a number — they are three different questions, decided at instrumentation time. Choosing wrong does not make the dashboard ugly; it makes the question permanently unanswerable, because the data you needed was never recorded.

A counter only goes up, which makes the raw value almost useless and its slope almost everything. The two things that go wrong: graphing the total instead of the rate, and mishandling the reset that happens every time the process restarts.

A gauge reports whatever the value was at the instant of the scrape. That is exactly right for queue depth and resident memory, and exactly wrong for anything that spikes — because a spike shorter than the scrape interval can leave no evidence that it ever happened.

A histogram stores counts per bucket instead of individual observations, which is what makes fleet-wide percentiles possible at all. The two decisions that determine whether it is useful: where you put the bucket boundaries, and whether you understand that every percentile it reports is an interpolation.

Five requests at 50, 55, 52, 48 and 3000ms have a mean of 641ms — a number no single request experienced. The mean is the wrong summary for latency because one extreme value drags it away from everything, and it cannot distinguish "everyone is slightly slow" from "one user in a hundred is unusable".

p50 describes the typical user, p99 describes the worst-served one percent, and the distance between them describes the system. The two errors that matter: reading a percentile without knowing the traffic volume behind it, and assuming percentiles compose across a call chain. They do not.

Cardinality is the product of every label's distinct value count, and it multiplies. One `user_id` label turns a three-series metric into three million, and the first thing that breaks is the monitoring system you were relying on to tell you what broke.

A label set is a schema: bounded value sets, names that mean the same thing in every service, and a migration path for the day you need to change one. Get it wrong and you either cannot join across services or cannot afford the series you created.

A log line is either a sentence a human greps or a record a program queries. The difference decides whether "how many payment timeouts hit provider X in the last hour" takes ten seconds or an afternoon of regex archaeology.

Nothing in any specification says what `warn` means. What it means is whatever your team decided, written down or not — and when it was never written down, everything becomes `info`, the error rate becomes unmeasurable, and the level field stops carrying information.

Without a shared identifier, logs from five services are five unrelated piles sorted by time. With one id propagated through every hop — and stored in a dedicated field — they become one request's story, and the log line becomes a doorway into the trace.

Log storage has a wider read audience, a longer retention and weaker access controls than the database the data came from. A token logged once is a token in a search index, in backups, and in whatever third-party service ships your logs — and no rotation policy knows it is there.

Log cost scales with traffic while its debugging value does not — the ten-thousandth identical success line teaches nothing. Sampling is how you keep the value and drop the volume, and the rule that makes it safe is simple: never sample what you would need during an incident.

Metrics tell you the endpoint got slower. A trace tells you which of the eleven things it touched got slower. One request, one timeline, every hop measured — and usually one span holding 80% of the budget that nobody suspected.

A span is a timed operation with a parent, a status and a bag of attributes. Which facts belong in attributes, which belong in span events, and which belong in a metric instead is the difference between a trace you can query and a very expensive log line.

Nesting is a claim about causality and containment: a child span asserts its parent was waiting for it. Get that wrong — most often by making a queued job a child of the request that enqueued it — and the waterfall stops describing anything real.

Trace context travels in-band with the work: a header on the HTTP call, a field on the queue message, an argument to the job. Every hop that forgets to carry it cuts the trace in half — and the caller looks like it was idle for 400 ms.

Six shapes cover most of what a waterfall can tell you: the staircase, the comb, the fat leaf, the gap, the overhang and the cliff. Learning to recognize them turns trace reading from scrolling into diagnosis.

In a fan-out, only the slowest branch controls when the request finishes. Optimizing any other branch produces a beautiful graph in your dependency dashboard and zero improvement for users — until the critical path moves, and then a different branch matters.

One query to fetch the users, then one query per user to fetch their orders. Every individual query is fast, every dashboard is green, and the endpoint takes 268 ms because it made 101 round trips instead of 2.

At 10,000 requests a second, tracing everything is a second production system. Sampling is inevitable; the question is whether you keep a random 1% — which discards almost every slow and failed request — or keep the ones that matter.

The trace says `pricing-service` spent 240 ms and has no children. That is where tracing stops and profiling starts: one tells you which process is expensive, the other tells you which function inside it is.

A CPU profile ranks functions two ways, and confusing them wastes afternoons. Total time says "this subtree is expensive"; self time says "this function is expensive". Only one of them tells you where to put the fix.

Width is time, height is stack depth, and the horizontal axis is not time at all. Getting that last part wrong is the single most common flame-graph misreading, and it makes people look for patterns that cannot exist.

Memory that is allocated and immediately freed never shows up as growth, so leak hunting finds nothing. It still costs: every megabyte allocated is a megabyte the collector must eventually walk, and at 500 MB/s that is where your latency went.

The first fork in every performance investigation. A CPU-bound service wants better algorithms or more cores; an I/O-bound service wants concurrency, batching or a faster dependency. Applying either fix to the other problem reliably makes things worse.

Profiling during an incident means capturing a baseline you do not have, on an instance that may be healthy, after the pathology has passed. Continuous profiling makes the baseline a query — and turns "did this release get slower" into a diff.

The dashboard says 120 ms and users say it is slow. Both are right: "the latency" was never one number. Response time decomposes into service time and wait time, and almost every production surprise lives in the waiting.

A one-in-a-hundred slow response sounds harmless until a page makes 40 calls, a user makes 30 page views, and every dependency has its own one-in-a-hundred. Rare events compose, and at scale the tail becomes the typical experience.

A latency target is only useful once it is divided. Give every hop an allowance, add them up, and the conversation changes from "make it faster" to "the payment provider is spending 45% of our budget and we have 15 ms left".

"We handle 10,000 requests per second" is not a capability claim until you say at what latency and with what error rate. Push a system to its maximum throughput and you will find the maximum is a place nobody wants to operate.

Concurrency equals throughput times latency. Three lines of arithmetic size a connection pool, expose an impossible capacity claim, and turn a queue depth into a wait time — which is most of what the law is for.

Load rises 20% and latency rises 400%. Nothing errored, no code changed, no dependency degraded. A queue formed — and queues turn a linear increase in arrivals into a non-linear increase in waiting.

Utilization says how busy a resource was. Saturation says how much work could not be served immediately. A CPU at 55% with twelve threads waiting for eight cores is not half idle — it is oversubscribed, and only one of those two numbers says so.

Accepting every request that arrives feels generous and produces the worst possible outcome: everything is slow, everything times out, and the capacity is spent on work nobody is still waiting for. A limit is a latency control, not just a safety valve.

A timeout is a statement about how long a caller will wait before deciding the answer is worthless. Set it too short and you manufacture load; too long and you tie up capacity waiting for work that stopped mattering minutes ago.

A CPU number without a denominator is not a measurement. Sixty percent of how many cores, against which cgroup quota, counting which of user, system, iowait and steal — and is anything actually waiting for a core?

Throughput stops rising, latency bends upward, and the run queue grows. Confirming CPU as the constraint takes three readings; the causes range from an O(n²) loop to logging in the hot path to a lock everything spins on.

An O(n²) loop over a collection that grew is a CPU bottleneck that scales with data, not traffic — which is why it passes load tests and fails in production. And the counterpoint: Big-O does not price cache locality, branch prediction or constants.

The memory number everyone watches is usually the wrong one. Resident, virtual, heap, cache and cgroup working set answer different questions, and allocation rate — the one nobody charts — often matters more than any of them.

Stable workload, rising memory, and a sawtooth of OOM restarts. Confirming a leak takes a trend under steady load; finding it takes two heap snapshots and a diff of what is still reachable.

Both grow, both end in an OOM kill, and they need opposite fixes. Three questions separate them: does the growth correspond to data you would use again, is there an eviction policy, and does usage stabilize?

Three numbers that people use interchangeably and should not: latency per operation, bytes per second, and operations per second. Plus the one that dominates write-heavy systems and appears on no dashboard by default — fsync.

Connection setup can cost more than the request it carries. RTT, bandwidth, retransmits, handshake counts and pool waits each answer a different question — and the first one to answer is whether the network is involved at all.

Nine numbers all get reported as "the database is slow" and they mean completely different things. Query duration measured at the application, split by statement, is the one that confirms it — and database CPU is the one that misleads most often.

Capture the statement, read the plan against reality, find where the estimate diverged, then decide which layer the fix belongs to — index, query, schema or application. Adding an index before reading the plan is guessing with extra steps.

The planner chooses a sequential scan over an index for good reasons: selectivity, table size, cache residency and the cost of random page access. Forcing the index because "indexes are fast" is the most confidently made wrong optimization in database work.

Nested loop, hash join and merge join are each optimal somewhere and catastrophic elsewhere. The planner picks one from a row estimate, so a wrong estimate does not make the query slightly slower — it makes the engine choose an algorithm built for a different problem size.

The database is 20% busy and every request takes four seconds. Nothing is overloaded — transactions are standing in line for the same rows. This is the shape that defeats capacity-based reasoning, because adding hardware makes the queue longer, not shorter.

A hundred concurrent requests, twenty connections, eighty in line. The database is 35% busy and every trace blames it, because the pool wait happens inside the span labelled "database" and outside anything the database can measure.

Replicas turn read capacity into a purchase, and the price is time. Lag is not a failure until the application assumes it is zero — and every read-after-write bug in a replicated system is that assumption meeting reality.

Hit rate is a ratio, and the thing that hurts you is a volume weighted by cost. The right question is never "how high is the hit rate" — it is which objects miss, how expensive each miss is, and how much load the misses put on whatever is behind the cache.

One popular key expires and ten thousand concurrent requests discover the miss simultaneously. Each one dutifully queries the database to repopulate it. The database receives ten thousand copies of the same query, and the cache that was protecting it becomes the mechanism that overloads it.

Sharding distributes keys, not traffic. One product goes viral, forty percent of requests land on one key, and the node holding it saturates while the cluster reports comfortable average utilization across every other node.

Arrival rate, processing rate, depth, oldest-message age, retry volume and dead-letter volume. Depth is the number everyone graphs and the number that explains the least; the rate pair tells you whether you are falling behind, and age tells you whether a human is already suffering.

10,000 jobs/s arriving, 8,000/s processed, backlog growing at 2,000/s. The gap is arithmetic, not opinion — and there are exactly four things you can do about it. Time-to-drain is the number to put in the incident channel.

A million tiny jobs and ten thousand hour-long jobs produce wildly different dashboards from the same word, "backlog". Depth measures accumulation; oldest-message age measures how long a human has been waiting. Only one of them belongs on a pager.

A dependency gets slower, clients retry, the retries become load, the dependency gets slower still. The feedback loop is what turns a 5% error rate into an outage — and it is the one source of traffic you can switch off yourself.

Every worker is occupied and the queue is 500 deep, so the obvious move is more workers. Whether that helps depends entirely on what the workers are busy *doing* — and if they are waiting on a shared dependency, adding workers makes things worse.

A collector trades pause time against throughput against memory footprint, and no tuning flag escapes the triangle. The lever you actually control is not the collector — it is how much garbage your code produces per request.

A single-threaded event loop runs one callback at a time. A 200ms JSON parse does not just make that request slow — it delays every other pending task by 200ms, including the health check that is about to fail.

Serialization, allocation and shape changes dominate real server-side JavaScript cost far more often than algorithmic choices. The engine optimizes aggressively for predictable code and deoptimizes quietly when you surprise it.

CPython pays a per-operation interpreter cost that no algorithm change removes, and its global lock means CPU-bound threads do not run in parallel. Neither fact makes Python slow at the thing most services actually do, which is wait.

No collector means no pauses and no free lunch: cost moves to allocator behaviour, fragmentation, and copies the language will make for you silently. And on modern hardware, where your data sits usually matters more than how many instructions you execute.

A JIT-compiled runtime starts interpreted and speeds up as it observes what the code actually does. That makes early requests slower, benchmarks without warm-up meaningless, and freshly-scaled instances a source of tail latency nobody attributes correctly.

Your p99 is 80ms and users still call the app slow. Server time is one line item in a budget that also contains DNS, TLS, render-blocking CSS, JavaScript parse and execute, and an image decode on a phone three years older than your laptop.

LCP asks "did anything useful appear?", INP asks "did the page answer when I touched it?", CLS asks "did it move while I was reading?". They are proxies for three different user frustrations — and they are web-platform definitions that have already changed once.

HTML discovers CSS, CSS blocks the paint, script blocks the parser, and the image nobody prioritised is fetched last. The waterfall shows a dependency chain — and the difference between a resource being downloaded and the page being usable.

A 400KB bundle is not one cost. It is downloaded, parsed, compiled and executed — and gzip only helps with the first of those. The last three are CPU on a device you did not choose and cannot upgrade.

Images are usually most of a page's weight and rarely the thing holding up interaction. They are also where the cheapest wins live — serving a 3000px photo into a 400px slot is a mistake that costs nothing to fix and shows up immediately in LCP.

One thread runs your JavaScript, computes layout, paints, and handles the user's tap. A 300ms task anywhere in that list means a 300ms wait everywhere else in it — which is why "the page freezes when I scroll" and "my handler is slow" are the same bug.

An in-process function call costs nanoseconds and either returns or throws. The same call across a network costs milliseconds, serialises both ways, waits in three queues you cannot see, and has a third outcome: no answer at all.

Call seven services in parallel and wait for all of them, and your latency is not the average — it is the maximum. A dependency that is slow one time in a hundred becomes a request that is slow seven times in a hundred, which is how p99 problems become p93 problems.

Four dependency calls take 740ms in a chain and 300ms fanned out. The parallel version is not simply better: it triples the instantaneous load on everything downstream and turns one failure into four things to reason about at once.

Light in fibre travels about 200,000 km/s. Frankfurt to Virginia and back is roughly 13,000 km of that, so no amount of tuning gets a round trip under about 65ms — and a request that crosses the Atlantic four times has spent a quarter of a second before doing any work.

A link that drops one packet in a thousand looks almost perfect on an average-latency graph. What it actually does is give one request in a few hundred an extra couple of hundred milliseconds, which is invisible at p50 and dominates p99.

Every guarantee that several machines agree on something is paid for in round trips. A quorum write is at least one; consensus is more; a distributed lock is two plus however long the holder keeps it. The guarantee is often worth it — the cost is never zero.

Average traffic, a peak multiplier, per-request cost and a latency target become a instance count through five multiplications — each one an assumption you can name, challenge and re-measure. The output is an estimate, and saying so is what makes it useful.

Running at 100% utilization means every burst, every failed instance and every deploy becomes an incident. How much headroom is a real decision with a real cost, and the number comes from burst shape, scale-up time and blast radius — not from a convention.

An autoscaling policy is a claim about what your bottleneck is. Scale on CPU and you have claimed the service is CPU-bound; when it is actually waiting on a database, the policy never fires while users time out.

Between a traffic spike and a new instance serving real traffic sit five delays: the metric window, the evaluation interval, provisioning, boot, and warm-up. Add them up honestly and you often find the spike ends before the capacity arrives.

Compute, database, cache, bandwidth, third-party calls and inference add up to a number per request. Track it next to latency and a whole category of "optimizations" reveals itself as buying a small latency win with a large permanent bill.

Capacity asks how much load the system can take. Efficiency asks how much resource each unit of work consumes. A system can scale beautifully while wasting most of what it buys — and the two problems have different fixes, different costs and different urgency.

Baseline, load, stress, spike and soak are five different tests answering five different questions. Most load tests fail before they start — against a warm cache, an empty database and one hot key, they measure a system that does not exist.

Ramp, spike, step and soak are not stylistic choices. Each shape exposes a different failure: ramps find the knee, spikes find scaling lag and cold starts, soaks find leaks and drift. Choosing a shape is choosing what you are willing to find out.

A load generator that waits for each response before sending the next one stops sending requests exactly when the system stalls. The requests that would have been slowest are never issued, never measured, and the reported p99 can be an order of magnitude better than what users experience.

A benchmark is an experiment, and most benchmarks fail as experiments before they fail as measurements. Warm-up, environment, workload realism, repetition, variance and a baseline are the difference between a number you can act on and a number you can quote.

Different environments, no warm-up, unrealistic payloads, averages without variance, several variables at once, and measuring something the system never actually does. Each produces a decisive number, and each is a reason to refuse to act on it.

The function got 40% faster and the request did not. A microbenchmark measures one operation in isolation; an end-to-end benchmark measures the system with its contention, I/O and queueing. Each is misleading when asked the other's question.

p95 moved from 180 ms to 260 ms. Before filing the bug, establish that both numbers answer the same question: same traffic mix, same data, same environment, enough samples. Then compare the difference against the noise you already know your measurement has.

An SLI is a ratio: good events over valid events. The hard parts are not the arithmetic — they are deciding what counts as good, what counts as valid, and where in the request path you measure, because each choice moves the number by more than most outages do.

An SLO is an SLI plus a target plus a window: "99% of checkouts complete under 300ms over 28 days". The target is not an aspiration — it is a commitment about how much unreliability you are willing to pay to avoid, and 100% is always the wrong answer.

An SLA is a contract: a reliability promise to a customer with a defined consequence when you break it. It is deliberately looser than your internal SLO, it is written by people who are not on call, and alerting on it means you find out you owe refunds at the same moment as your legal team.

If the objective is 99.9%, then 0.1% of failure is not a defect — it is a budget. Treating it as something to spend rather than something to avoid turns every reliability-versus-velocity argument into an arithmetic question, which is the only version of that argument anyone ever wins.

An alert is a claim that a human should stop what they are doing right now. `CPU > 80%` is not that claim — it is a fact about a machine that may or may not correspond to a user having a bad time. Alert on symptoms users feel; keep causes on dashboards where they belong.

Alert fatigue is not a morale problem, it is a detection failure. Every page that turns out to be nothing raises the probability that the next real one is acknowledged late, investigated slowly, or dismissed entirely — and the arithmetic that produces it is measurable.

Alerting on "error rate above 1%" picks a threshold with no relationship to what you promised. Alerting on burn rate asks a better question: at the current failure rate, how long until the error budget is gone? Fast burn pages, slow burn tickets, and two windows each stop the alert from lying.

A dashboard is not a place to put metrics — it is a tool for answering a specific question under time pressure. Two hundred charts is not thoroughness, it is an unindexed archive, and at 03:00 the difference between eight charts in the right order and two hundred in no order is the difference between five minutes and fifty.

Mitigation and diagnosis are different jobs, and doing them in the wrong order costs users minutes they never get back. Stabilize first, then form a hypothesis you can disprove in two minutes instead of browsing dashboards hoping something looks odd.

Two charts moved together at 14:03. So did four others. Establishing that one caused another needs a mechanism you can state, or an intervention you can run — and during an incident you usually have time for exactly one of them.

The timeline shows database latency rising at 12:08 and API p99 rising at 12:10. That ordering is a property of your alert thresholds and scrape intervals as much as of the system — and the first thing you observed is routinely not the first thing that happened.

The highest-yield first question in any performance incident is what changed, and it is only answerable in seconds if changes appear on the same time axis as the metrics. Code deploys are the easy part; config pushes, feature flags and someone else's release are the ones that leave no mark.

You removed the CPU bottleneck and the system is still slow — because the constraint moved to the database, where it had been hiding behind the CPU limit all along. This is what success looks like, and predicting the next constraint is what separates a plan from a sequence of surprises.

Caching buys database load and sells freshness. Compression buys bandwidth and sells CPU. Batching buys throughput and sells latency. There is no move that is purely faster — and the ones that appear to be are usually selling reliability quietly.

Every one of these is a plausible move that a competent engineer makes under pressure, and every one shares a single property: no measurement before, or no measurement after. That is the tell, and it is the only thing they have in common.

An agent run is a chain of network round trips nobody wrote explicitly: model call, tool call, model call, tool call. The wall clock is dominated by that chain, so step count is the variable that matters most and the one least likely to appear on a dashboard.

A model call is three different waits with three different causes. Provider scheduling you cannot control, time to first token that scales with your prompt, and generation that scales with your output — and only two of those are yours to shorten.

Model 2.2s, search 0.8s, model 1.6s, database tool 0.2s, model 1.4s — 6.2 seconds in a straight line. The question a waterfall answers is which of those steps are sequential because the data requires it, and which are sequential because that is the order the model happened to emit them.

Cost per run is tokens times price times steps — and the tokens term grows every step, because each tool result is appended to a context that every subsequent call must pay for again. That quadratic-ish growth is why long runs cost far more than their step count suggests.

Rate, errors and duration all look healthy while the agent confidently tells a customer something untrue. Traditional service metrics measure whether the machinery ran; agent systems need metrics for whether the task was actually accomplished.