9 lessons

Observability Fundamentals

Understanding internal behavior from external signals. The measure-before-optimizing loop, what each signal type is actually good at, the golden signals, RED and USE, and how instrumentation reaches a backend at all.

Symptom→Signal→Measurement→Hypothesis→Evidence→Root Cause→Change→Validation→Regression Check

Every lesson below starts from an observable symptom and ends with the measurement that proves the fix worked. Numbers carry a label saying whether they were measured, estimated, simulated or invented to show a shape.

Observability Is Not a Dashboard

Monitoring answers the questions you thought to ask when you built the dashboard. Observability is whether you can answer a question nobody anticipated — without shipping new code first. The test is not how many tools you run; it is what you can ask at 03:00.

Symptom · Something is clearly wrong — customers are complaining — and every dashboard is green. The only way forward anyone can suggest is to add logging and redeploy.

Measure Before You Optimize

▶ lab

The single most expensive habit in performance work is proposing a fix before taking a reading. This is the loop that replaces it: Problem → Measure → Locate → Understand → Change → Measure Again — and the six questions that turn "it is slow" into a specific reading at a specific layer.

Symptom · Within five minutes of "the API is slow", someone has proposed adding Redis, bumping the instance size, or increasing the thread pool — and nobody has yet looked at a number.

From Symptom to Root Cause

The diagnostic chain — Symptom → Signal → Measurement → Hypothesis → Evidence → Root Cause → Change → Validation → Regression Check — and the discipline that makes it work: write the hypothesis down before you look, so you can be wrong out loud instead of quietly.

Symptom · An investigation that has been running for an hour, has looked at fourteen dashboards, and cannot state what it currently believes or what would disprove it.

Metrics, Logs, Traces, Profiles

Four signals, four different questions. Metrics tell you something changed; traces tell you where the time went; logs tell you what exactly happened; profiles tell you what the CPU was doing. No single one explains an incident, and knowing which to reach for first is most of the speed.

Symptom · An investigation that reaches for logs first, greps 40GB, and finds nothing — because the question was "which service consumed the time", and logs are the wrong instrument for that question.

The Four Golden Signals

Latency, traffic, errors and saturation. Four numbers that describe almost any request-serving system well enough to know whether it is healthy and, when it is not, which direction to look. The value is not the list — it is that the four are read together.

Symptom · A service with either no dashboard or forty panels, where nobody can answer "is it healthy?" in ten seconds.

RED: Rate, Errors, Duration

Three numbers per request-handling service: how many, how many failed, how long they took. RED is the fastest way to make every service in a fleet legible in the same shape — and it goes blind the moment work stops being request-shaped.

Symptom · A fleet of thirty services where each team invented its own dashboard, so comparing two services during an incident means learning two vocabularies under pressure.

USE: Utilization, Saturation, Errors

For every resource, ask three questions: how busy is it, is work waiting for it, and is it failing? The middle question is the one that matters and the one most dashboards omit — which is why "CPU is only 40%" keeps getting offered as evidence that CPU is fine.

Symptom · Latency is high, every utilization chart looks moderate, and the team concludes there is no resource problem — while requests queue for a pool nobody charted.

Instrumentation: From Code to Signal

Telemetry does not appear; it is emitted by code, batched by a client, shipped to a collector and stored by a backend — and every hop can drop data, add latency or cost money. Knowing the path is what lets you trust the dashboard, and notice when it lies.

Symptom · A dashboard that went flat during the incident — and nobody can say whether traffic stopped, or the telemetry pipeline did.

OpenTelemetry Concepts

A vendor-neutral way to describe traces, metrics and logs, propagate context across process boundaries, and ship the result anywhere. Worth understanding as a set of concepts — signals, context, semantic conventions, collector — rather than as a product to install.

Symptom · Six services in four languages, each with its own tracing library, its own attribute names and no shared trace id — so a cross-service request cannot be reconstructed at all.