DebuggabilityGENERALSCALE-SPECIFICCONTESTED
Debuggability by Design
A system should be able to answer what happened, why, for which request and user, on which version, and from which state. None of those are answerable later if the design did not record them.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind survives until the requirement changes.
The question
Six weeks from now, someone asks why this customer was charged on the 3rd. What has to have been designed in for that to be a ten-minute answer rather than a day?
The requirement
Support escalates: a customer says they paused on the 1st and were charged on the 3rd. Two engineers spend a day and produce a theory nobody can confirm, because the only record of the subscription is its current state, which has since changed twice.
The obvious build
Add logging when we need it. Debugging is what logs are for, and adding a log line takes ten seconds — there is no reason to design for it up front. This is right about the ten seconds and wrong about when they are available.
Why it breaks
The ten seconds are available now and the question arrives in six weeks, about an event that happened three weeks ago. Log lines are not retroactive, which makes "add it when we need it" a plan that never fires (The Cost of Change).
How it breaks as requirements change
- The ten seconds are available now and the question arrives in six weeks, about an event that happened three weeks ago. Log lines are not retroactive, which makes "add it when we need it" a plan that never fires (The Cost of Change).
- The second requirement is "which of the two pause paths did this go through", and the answer is not in the code's current state either, because the admin path was refactored since.
- The state-only design cannot distinguish "was never paused" from "paused and resumed", so the two most likely explanations are indistinguishable in the data (Explicit State).
- Adding a customer id to log lines later is easy; making the ones written last month contain it is not. Every debuggability decision has this shape, which is why it belongs in the design and not in the follow-up ticket.
- Meanwhile the logs that do exist are unusable: forty thousand subscriptions produced one line, and the per-subscription logging was removed for cost rather than replaced with something cheaper (The Log Bill and What It Is Buying).
Requirement→Constraints→Invariants→Responsibilities→Boundaries→Interfaces→State→Dependencies→Failure→Implementation→Tests→Feedback→Evolution
What limits the solution, and what must never stop being true
This domain leads with these two. A design that ignores its constraints is not a design, and an invariant nobody named is one nothing is protecting.
Constraints
- The charge itself is real and irreversible; the question is not whether to refund but whether it will happen again tomorrow (The Production Loop).
- Logs are retained for seven days and the incident is about something that happened on the 1st.
- The nightly billing job processes forty thousand subscriptions and logs one line per run, because logging one line per subscription was costing more than the database.
- Nothing can be added retroactively. Whatever was not recorded on the 3rd does not exist.
Invariants
- Every state change that a customer can see is reconstructable after the fact, from a record made at the time and not from current state.
- Every record of a decision names the version of the code that made it, because "it works on main" is not a claim about the 3rd.
- Any single customer-visible event can be traced back to the request that caused it, using one identifier the customer or support can supply (Stable Identifiers).
- Nothing recorded for debugging contains a secret or a piece of personal data that the deletion path does not reach (Sensitive State).
Who owns what, and where the seams fall
Responsibilities decide boundaries; boundaries decide what an interface has to say.
Responsibilities
- The domain owns emitting transitions as facts. A state change that is only a column update has destroyed the information that a change happened at all.
- The application layer owns attaching identity: which request, which actor, which tenant, which version (Request Context Propagation).
- The deployment pipeline owns making the version knowable — a build id in the record, not a guess from a deploy timestamp ("What Changed?" — Deploy Markers and the Invisible Deploys).
- The Observability domain owns what happens to the signals after they are emitted: sampling, retention, cardinality and cost. This domain stops at whether there was anything to emit.
Boundaries
- The boundary that matters is the state transition. Deciding that transitions are things rather than side effects of an update is what makes the history exist (State Machines).
- The second is the request boundary, where identity is attached once and carried, rather than passed by hand and forgotten in the third call down (Stable Identifiers).
- A hard line runs between debugging records and business records. An audit trail that finance depends on is a product feature with retention rules; a debug log is not, and conflating them means either finance loses data at seven days or the debug log has to be kept for seven years.
Five questions, and what answers each
Every production question reduces to one of five, and each is answered by a different piece of recorded data. The point of listing them is that the pieces are unrelated: a system can answer three of them perfectly and be useless because the fourth is missing.
The version question is the one most often absent and most often decisive. A behaviour change on a specific date is a deploy question before it is a logic question, and answering it in seconds rather than hours is the difference between a build id in the record and a search through deploy history.
- What happened — a transition emitted as a fact, so that "it changed" survives the next change (Explicit State).
- Why — the decision and the inputs it saw, not only the outcome. The billing job's "decision=skip, reason=paused" is the whole answer.
- For whom — one identifier that support can supply, reaching every record (Stable Identifiers).
- Which version — a build id in the record, because a config change and a code change look identical from the outside (Change Correlation).
- From which state — the state at decision time, which current state no longer tells you.
UserLLMAgentToolDataDecisionHumanGuardrail
Cheap now, impossible later
SIMULATEDThe cost estimates are Engineer Atlas model figures over a teaching codebase, not measurements from a real team; treat the ordering and the asymmetry as the content and the durations as illustrative. What transfers is the shape — each row is minutes-to-hours before and unbounded-or-impossible after — and that shape is what the argument rests on, not the numbers.
This is the whole economic argument of the module, and it is unusually clean: almost every item here costs minutes during feature design and is either expensive or literally impossible to add afterwards, because the data for the period in question was never written.
The right-hand column is not exaggeration. "Impossible" means exactly that — a question about the 1st, asked on the 21st, against records that never carried the field. You can add it going forward, and the incident you have is still unanswerable.
| Decision | Cost if made during design | Cost after the first incident |
|---|---|---|
| Transitions recorded as events | A table and one write per change; roughly an hour. | A table, a decision about backfill, and no history before today. The incident stays unanswered. |
| Correlation id from the edge | A middleware and a context parameter; an afternoon. | Every log line already written lacks it. Threading it through later touches every layer, and old records never gain it (Correlation IDs: Turning Lines Into a Story). |
| Build id on every record | One environment variable and one field; minutes. | Cheap to add and still leaves you unable to say what was running on the 3rd. |
| Decision logged with inputs | One structured line at the branch that decides; minutes. | Requires finding the branch, and any theory about past runs remains a theory (From Symptom to Root Cause). |
| Injected clock | A parameter; minutes if done first, a refactor if not (Time as a Dependency). | A refactor of the call graph, usually deferred forever, so the bug stays unreproducible. |
| Sensitive fields typed as sensitive | A wrapper type; an hour. | A grep for field names, a log-vendor deletion request, and a compliance conversation (Sensitive State). |
How the answer goes missing
Each row below is a system that has logging, monitoring and an audit table, and still cannot answer the question. That combination is the normal case, which is why "do we have logs" is the wrong question to ask during design.
The pattern across all of them: the data exists but is not addressable by the thing support can supply, or it describes what the state is rather than what the decision was. Those two failures cover most unanswerable incidents.
Why an incident stays a theory
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Support has an email address; the logs have a UUID. | Forty minutes to find the customer before debugging starts, and it is done by hand each time. | No stable identifier connects the human-facing entity to the recorded one (Stable Identifiers). | Mint the id at the edge and make it the key support is trained to ask for. |
| The row says paused; the customer was charged. | Both facts are true and there is no way to know which came first. | Current state was stored; the transition and its timestamp were not (Explicit State). | Emit transitions as events with the state they moved from. |
| The job logs "processed 40,000 subscriptions". | No way to tell what it decided about any particular one. | Logging was reduced for cost by removing detail rather than by logging decisions instead of rows (The Log Bill and What It Is Buying). | Log one line per interesting decision — skips, conflicts, errors — and a count for the rest. |
| Behaviour changed on the 3rd, code did not. | Hours spent reading logic that was never wrong. | A config or flag change carried no marker, so it is invisible in the timeline (A Config Change Is a Production Change). | Treat config changes as deploys with their own markers ("What Changed?" — Deploy Markers and the Invisible Deploys). |
| The bug depends on the time of day it ran. | Cannot reproduce locally; three attempts to fix it by inspection. | now() inside the rule, so the decision cannot be replayed with the inputs it had (Time as a Dependency). | Inject the clock and record the instant used in the decision line. |
| The audit table has the answer and it is wrong. | The whole investigation runs off a bad premise for half a day. | The audit path is never exercised in tests and diverged from the real path months ago. | Assert the record in the same test that asserts the behaviour, so the two cannot drift (Testing as Design Feedback). |
How to build it
Most important first.
- Decide the five questions before implementing: what happened, why, for whom, on which version, from which state. Write the query you would run to answer each one — if you cannot write it, the data does not exist (From Symptom to Root Cause).
- Record transitions as events, not as column updates. "paused_at was set" is a state; "PauseStarted at 09:12 by actor X for reason Y from state active" is an answer (Explicit State).
- Log the decision, not just the outcome. The billing job's useful line is not "charged 40,000" but "subscription 8842: state=active, renewal=today, decision=charge" — one line per *interesting* case, not per row (Logging at Boundaries).
- Attach one id at the edge and propagate it, so a customer-supplied reference reaches every log line, every event and every downstream call (Correlation IDs: Turning Lines Into a Story).
- Put the build id in every record. A change in behaviour on the 3rd is a deploy question first and a logic question second, and the deploy question is answerable in seconds if the version is in the row (Change Correlation).
- Keep the domain core deterministic, so that having the inputs means being able to reproduce the decision rather than merely reading about it (A Deterministic Core).
What the next change costs
The field this whole domain exists for. A structure is only better if it makes the change after this one cheaper — and it is worth saying which changes it does not help.
Cost of the next change
- Designed in: answering a new question — "which pause path did this take" — costs one field on an event that already exists, and it works for everything after the deploy.
- Retrofitted: the same question costs a new event type, a decision about backfilling, an admission that events before today have no answer, and usually a second incident before the gap is noticed.
- The next feature costs almost nothing, because identity propagation and transition recording are already in place and the new feature inherits them. That is the compounding return and it is why this is a design decision rather than an operational one.
- What stays expensive forever: questions about the past. No design makes last month's missing data appear, which is the one asymmetry that justifies deciding this before shipping rather than after the first incident.
What the recommended approach costs
- Every record costs storage, write throughput and money, and the first thing an infrastructure cost review cuts is the logging that has not yet paid for itself. Designing for debuggability means defending it in a budget conversation you will not win with "it might help".
- Transition events are a second representation of state, and two representations can disagree. Deriving current state from the events avoids that and costs a read; keeping both is faster and needs reconciliation (Materialized Views: A Read Model That Lags).
- Making the version and actor available everywhere means threading context through code that does not otherwise need it, and that thread is real coupling with a real cost (Request Context Propagation).
What can go wrong
Failure modes
- Everything is logged, so nothing is findable and the bill is large. Volume is not the property being optimised; answerability is (The Log Bill and What It Is Buying).
- The transition record exists and is not queryable — a JSON blob in a text column, keyed by nothing, that requires a full scan to ask "which subscriptions paused on the 1st".
- Debug records accumulate personal data, and then the deletion request arrives and the records are not in the deletion path (Sensitive State).
- The version is recorded and the deploy that mattered was a config change, which carries no build id and therefore no trace (A Config Change Is a Production Change).
- The mitigation fails too: an audit table is added and nobody reads it, so it is never validated. It is then trusted absolutely during an incident, at which point its three-month-old bug becomes the theory everybody works from.
Dependencies, and their direction
- Debuggability depends on identity existing before it is needed, which is why Stable Identifiers comes before everything else in this module.
- It depends on the deployment system exposing a version at runtime — an environment variable set by the pipeline, not a hand-maintained constant (Tags Versus Digests).
- It depends on retention outliving the questions. A seven-day retention against a monthly billing cycle guarantees that every billing question is asked about data that no longer exists, and no amount of logging fixes that (The Log Bill and What It Is Buying).
- The domain core should depend on none of it. A rule that takes a logger is a rule with an extra reason to change (Functional Core, Imperative Shell).
Misreads
- "So log more." Volume is the opposite of the goal. One line naming a decision beats forty thousand lines naming rows (Logging at Boundaries).
- "This is observability." Observability is what you do with the signals — dashboards, alerts, traces, sampling. This is whether the signals were possible to emit at all, and it is decided in the feature design (Observability Is Not a Dashboard).
- "An audit log covers it." An audit log records who did what for compliance. It rarely records why a job decided not to act, which is the most common debugging question and the one nobody thinks to record (The Audit Trail).
- "We can reproduce it locally." Not if the decision depended on the wall clock, a random value or a row that has since changed. Reproducibility is a design property, not a lucky one (A Deterministic Core).
Testing it, and how it ages
What to test, and at which boundary
- Assert that a state change emits a transition record with actor, from-state, to-state and reason. That is a behavioural assertion, not a logging assertion, and it belongs in the domain tests (What a Unit Is).
- Assert propagation: a request with a correlation id produces downstream records carrying the same id. This is the test that catches the third call down having dropped it (Where a Test Must Be Real).
- Assert redaction: a field marked sensitive cannot be serialised into a log record. Prefer making it a compile error over making it a test (Sensitive State).
- Run the actual incident query in a test against seeded data. "Can we answer this question" is testable, and testing it is what stops the answer from silently rotting.
How this design ages
- Transition records grow into the thing finance, support and analytics all query, at which point they stop being a debugging aid and become a product surface with a compatibility obligation (Backward Compatibility as a Constraint).
- Logs get sampled as traffic grows, and the sampling has to be designed so that errors and rare transitions are always kept while the common path is thinned. Uniform sampling drops exactly the lines an incident needs (Sampling Without Throwing Away the Evidence).
- It stops being sufficient once the answer spans services, at which point correlation stops being a field and becomes a propagated trace context (Distributed Tracing).
Where this applies
This domain's advice is contested more than most. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view rather than a caricature.
- GENERALThat a record cannot be created retroactively is a property of time rather than of technology, so the asymmetry holds everywhere. What varies is how much the platform gives you for free — a managed runtime may attach request ids and versions without being asked, which removes the mechanics but not the decision about what a transition is.
- SCALE-SPECIFICAt one server and a hundred users you can read the logs, and much of this is overhead. Past a few services and any real traffic, unaided reading stops working entirely and the questions can only be answered by structure that was designed in — the crossover is roughly where a person can no longer hold the request path in their head.
- CONTESTEDThe strongest opposing view is that debuggability is over-designed relative to its use: most systems answer most incidents from current state and a stack trace, and the transition tables, correlation plumbing and version stamping represent permanent complexity and cost against a handful of hard incidents a year that were mostly solved by reasoning anyway. That is a fair reading of low-stakes CRUD systems, and it collapses wherever the question is about money, where a plausible theory is not an acceptable answer.
Where the depth lives
This domain teaches the codebase-level structure and hands the rest off.
Performancewhat-observability-isstructured-loggingcorrelation-idsdistributed-tracingincident-debuggingsymptom-to-signal
Domains that do not exist yet
- — Testing & Reliability Engineering — an incident review asks whether the system could answer the question, and the answer is decided long before the incident.