Stable Identifiers
request_id, order_id, workflow_id. Correlation is a design decision made in the first week or not at all, because an id cannot be added to records that were written without it.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind survives until the requirement changes.
A customer says "it did not work". What single value do they, or support, hand me that finds every record of what happened?
Support wants to stop escalating "customer says pause failed" with a screenshot. They want a reference the customer can read off the screen that an engineer can paste into one query.
Use the database primary key. Every row has one, it is unique, it is already there, and it costs nothing — which is genuinely true for the single-entity case and is why this is where everyone starts.
A primary key exists only after the row is written, so anything that happens before the write — validation failures, timeouts, rejected requests — has no id at all. Those are exactly the cases support escalates (Designing the Happy Path Last).
- A primary key exists only after the row is written, so anything that happens before the write — validation failures, timeouts, rejected requests — has no id at all. Those are exactly the cases support escalates (Designing the Happy Path Last).
- It identifies an entity, not an operation. Three pause attempts on one subscription share a key, so "which attempt was this" is unanswerable from the key alone.
- It does not cross the boundary. The queue message and the nightly job have their own keys, so the operation fragments into three unrelated identities as it moves (Three Nodes, Three Logs, and You Cannot Sort by Timestamp).
- Sequential keys leak volume — an order id of 1043 tells a competitor how many orders you have had — and pressure to fix that later produces a second id, at which point there are two and code disagrees about which is canonical (Distributed Uniqueness: One Name, Many Shards).
- The retrofit is the expensive part: adding a request id later means a middleware, a context parameter through every layer, a client change, and the knowledge that every record written before today has none (Debuggability by Design).
What limits the solution, and what must never stop being true
This domain leads with these two. A design that ignores its constraints is not a design, and an invariant nobody named is one nothing is protecting.
- The reference has to be short enough to read over the phone and unique enough not to collide across a year of traffic.
- It crosses a browser, an API, a database, a queue and a nightly job, three of which are already in production and were not designed for it.
- It appears in logs held by a third party, so it must not be derivable from anything personal (What You Just Wrote Into a Log Half the Company Can Read).
- The mobile client cannot be updated on your schedule, so anything requiring the client to generate it has a six-week lag on half the traffic.
- One operation has one id for its whole life, across processes, retries and hops. An id that changes at a boundary correlates nothing.
- The id survives a retry: the same logical attempt keeps the same id, so a duplicate is visible as a duplicate rather than as two unrelated operations (Idempotency by Design).
- Ids are opaque. Nothing encodes meaning into them, because encoded meaning becomes a contract that has to keep being true (Primitive Obsession).
- An id never carries personal data, since it travels to places the deletion path does not reach.
Who owns what, and where the seams fall
Responsibilities decide boundaries; boundaries decide what an interface has to say.
- The edge owns minting the request id — one place, on the way in, before any handler runs (Request Context Propagation).
- The domain owns entity ids and workflow ids, because they are business identity and outlive any request. Generating them in a controller is how an id becomes tied to a transport.
- Every boundary owns propagation: an id that arrives must leave, on the outbound call, on the queue message, in the log line.
- Nothing owns interpreting them. The moment code parses an id for meaning, the format is a contract (API Stability).
- Three distinct scopes, and conflating them is the central mistake.
request_idlives for one HTTP request;order_idlives for the life of the entity;workflow_idspans every request and job belonging to one business operation (What Counts as the Same Operation?). - The propagation boundary is the process edge. Inside, an ambient context is acceptable; crossing out, the id must become an explicit part of the message, because ambient context does not survive a queue (Carrying the Trace Across the Gap).
- The customer-facing boundary needs a different id from the internal one: short, readable, non-sequential. That is a product decision as much as a technical one.
Three ids, three lifetimes
The most common mistake is not missing ids but having one where three are needed. A request id is useless for a question about an operation that spans a queue; an entity id cannot distinguish two attempts; a workflow id alone cannot tell you which HTTP call produced a particular log line.
They are cheap to have all three of and expensive to add later, which is an unusually easy trade. What is not easy is the propagation: every arrow in the pipeline below is a place an id can be silently dropped, and a dropped id produces a gap that looks exactly like an absence of activity.
- 1Browser
Generates a client attempt id and sends it as
Idempotency-Key; retries reuse it.fails by Generating a fresh id per retry, which makes a duplicate submission look like two customers.
- 2Edge
Mints
request_idfor this HTTP call; accepts the client id if well formed; puts both on the context.fails by Minting only after routing, so requests rejected by auth or validation have no id at all.
- 3Command
Mints
workflow_idfor the pause operation; stores it with the idempotency record.fails by Reusing
request_idas the workflow id, so the scheduled resume three months later belongs to a request that ended in 200ms. - 4Transaction
Writes the transition carrying
subscription_id,workflow_id,request_idand the build id.fails by Writing only
subscription_id, so the record cannot be tied to the request that caused it. - 5Outbox / queue
Copies both ids onto the message, because ambient context does not cross a process boundary.
fails by Relying on a thread local, which is empty in the consumer and produces an empty field rather than an error (Carrying the Trace Across the Gap).
- 6Nightly job
Mints its own
run_id, and logs the subscription'sworkflow_idfor any non-routine decision.fails by Logging only
run_id, so the job's decision cannot be joined to the pause that caused it. - 7Error screen
Shows the
workflow_id, shortened, as a reference the customer can read out.fails by Showing nothing, so support escalates a screenshot and an engineer starts with a timestamp and a guess.
The last row is the one that changes support's day and it is the one most often missing, because it looks like a frontend concern. It is a debuggability decision that happens to be rendered.
Typed, opaque, and generated where they are needed
Two decisions in the code below carry most of the value. Ids are distinct types, so passing a customer id where a subscription id belongs does not compile — that eliminates a class of investigation that otherwise ends with "we were looking at the wrong record for an hour".
And the entity id is generated in the domain rather than by the database, which means it exists before the write. The log line about the write that failed can name the subscription it was about, and that is precisely the line that is missing in the naive design (Designing the Happy Path Last).
1type Id<Brand> = string & { readonly __brand: Brand }2type SubscriptionId = Id<'subscription'>3type WorkflowId = Id<'workflow'>4type RequestId = Id<'request'>5 6type OpContext = {7 requestId: RequestId // this HTTP call8 workflowId: WorkflowId // this business operation, across processes9 actor: Actor10 build: string11}12 13// generated in the domain, before any write:14const id = newSubscriptionId(rng) // rng injected, see randomness-as-dependency15 16// crossing a process boundary: explicit, never ambient17await queue.publish({ ...msg, workflow_id: ctx.workflowId })18 19// findSubscription(customerId) does not compile. That is the point.The branded type costs a cast at the two edges where ids enter and leave as strings, and buys a compile error everywhere else. Where a language cannot express this cheaply, the same discipline needs a naming convention and a review habit, which is strictly weaker — worth saying plainly rather than pretending the technique is universal (Units in Names and Types).
The retrofit, priced
This is the clearest change-impact case in the module, because the difference is not effort but reachability: one column of the table is work, and the other column is work plus a permanent hole where the last year should be.
The cost line is worth reading carefully. Explicit context is not free — it puts a parameter in signatures that have no business caring about it, and that is a genuine coupling of every layer to the observability design.
A customer reports a failed pause. Support wants a single value from the error screen that returns the request, the transition, the queue message, the nightly job's decision and the email.
Nine modules, a queue message schema change, and a mobile release with a six-week tail during which half of traffic is uncorrelated. And every record written before the change stays unjoinable forever, so the incident that motivated the work is still unanswerable.
The ids are already on every record and every message. The change is displaying the workflow id on the error screen and teaching support to ask for it — an afternoon, and it works retroactively for every record already written.
How to build it
Most important first.
- Mint a request id at the edge for every request, including the ones that fail validation. Accept a client-supplied one if it is well formed, so a browser retry keeps its identity (Correlation IDs: Turning Lines Into a Story).
- Give long-running business operations their own
workflow_id, minted when the operation starts and carried by every request, job and event that belongs to it. This is the id that answers "what happened to my pause", which spans three processes. - Propagate through an explicit context object at boundaries rather than a global. A global works in a single-threaded process and fails silently the day work moves to a pool (Hidden Global State).
- Make ids opaque and typed. A
SubscriptionIdthat is not astringcannot be passed where aCustomerIdbelongs, which removes an entire class of debugging session (Units in Names and Types). - Generate entity ids in the domain, not the database, so the id exists before the write and the log line about the failed write can name it (Distributed Uniqueness: One Name, Many Shards).
- Put the id in the error the customer sees. A reference on the error screen is what turns "it did not work" into a query, and it is four lines of code (Validation Errors: Feedback, Not Verdicts).
What the next change costs
The field this whole domain exists for. A structure is only better if it makes the change after this one cheaper — and it is worth saying which changes it does not help.
- With ids designed in, adding a new component to the operation costs accepting a context parameter. It joins the existing trace for free, which is the whole return.
- Without them, the first correlation question costs a middleware, a context parameter threaded through every layer, changes to queue message schemas, a client release, and permanent blindness about everything before the change.
- Changing an id *format* later is close to the most expensive change in this list: stored ids, external references, customer-facing references, indexes and any encoded meaning all move together (Data Migration).
- What never gets cheaper: correlating records written before the id existed. That asymmetry is the reason this is a first-week decision (The Cost of Change).
- Explicit context propagation is a parameter threaded through functions that do not care about it, and that is real noise in every signature it touches. Ambient context avoids the noise and fails invisibly across async boundaries — this is a genuine choice with no free option.
- Typed ids cost a wrapper, conversions at the edges, and friction in every generic utility that expected a string. They buy a class of bug that is otherwise found in production.
- Random ids cost index locality; sequential ids cost information leakage and multi-writer coordination. Time-sortable schemes split the difference and leak coarse timing, which is sometimes exactly what you cannot leak (Distributed Uniqueness: One Name, Many Shards).
What can go wrong
- The id is minted per layer, so the same operation has three, and joining them requires a table nobody maintains.
- A retry mints a fresh id, so a duplicated operation looks like two independent ones and the duplicate is invisible in every dashboard (The Retry Is a Decision, Not a Reflex).
- Random v4 UUIDs are used as a clustered primary key on a high-write table, and insert performance degrades because every insert lands in a random page (An Index Scan Is Not Automatically Faster).
- Meaning gets encoded — a prefix for the region, a checksum, a date — and two years later a region is renamed and the ids are wrong or the parser is.
- The mitigation fails too: a workflow id is added and one job forgets to propagate it, so a subset of records silently drop out of every trace and the gap looks like "nothing happened" rather than "not recorded".
- Everything that logs, traces or emits events depends on the context carrying the id, which makes context propagation a cross-cutting dependency and one of the few genuinely justified ones (What Belongs in the Pipeline).
- The id scheme depends on a generator: sequential from the database, random UUID, or time-sortable. Each has a different index behaviour and the choice is hard to reverse once ids are stored (Allocation and Copies).
- Support tooling depends on the customer-facing reference existing, which means the error screen is part of the debuggability design and not a separate frontend concern.
- "A UUID everywhere solves it." Uniqueness is the easy half. The hard half is that the same operation keeps the same id across processes and retries, which is a propagation decision, not a generation one.
- "The trace id covers this." A trace id is scoped to one distributed trace and is typically sampled away. A workflow id spanning three days and four jobs is a different thing with a different lifetime (Trace, Span, Attribute, Status).
- "Ids should be human-readable." Readable at the support boundary, opaque everywhere else. Encoding meaning into the internal id is what turns a rename into a data migration (Primitive Obsession).
- "We can add correlation when we need it." You can add it going forward. The incident that made you want it is about last week, and last week has no id (Debuggability by Design).
- primitive-obsession
Testing it, and how it ages
- Assert that a request carrying a correlation id produces downstream records with the same id — including across the queue, which is where propagation is usually dropped (Where a Test Must Be Real).
- Assert that a request without one gets a generated id rather than an empty field, because an empty correlation field is worse than none: it joins every unrelated record together.
- Assert that a failed request still produces a record with an id. The happy path is not where correlation matters (Designing the Happy Path Last).
- Type-level, not test-level, where possible: passing a
CustomerIdwhere aSubscriptionIdis expected should not compile (Units in Names and Types).
- Ids outlive everything. They end up in customer emails, third-party systems, support macros and finance exports, at which point the format is a public contract with no version field (Backward Compatibility as a Constraint).
- Correlation ids grow into trace context once the system spans services, and a design that already propagates an explicit id has a short path to that; one relying on globals does not (Distributed Tracing).
- The scheme is forced to change when volume breaks it — a sequential id under multi-region writes, or a random one under an index that needs locality (Hash Partitioning and the Modulo Trap).
Where this applies
This domain's advice is contested more than most. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view rather than a caricature.
- GENERALThat records without a shared key cannot be joined is a property of data rather than of a stack, so it holds from a single process to a fleet. What varies is how much the runtime propagates for you: a platform with built-in request context removes the plumbing but not the decision about what a workflow is.
- SCALE-SPECIFICIn one process with one log file, the timestamp is an adequate correlation key and this whole lesson is overhead. It becomes load-bearing the moment there are concurrent requests interleaved in one stream, which is far earlier than most teams expect — roughly the first time two users act at once.
- LANGUAGE-SPECIFICAmbient propagation is idiomatic and reliable in Go with
context.Contextand in Java with thread locals under a synchronous model; it is a persistent source of silent loss in JavaScript across async boundaries and in any language with a work-stealing pool. The design is identical; the mechanism's failure rate is not.
Where the depth lives
This domain teaches the codebase-level structure and hands the rest off.
- — System Design — id generation schemes under multi-region writes are a coordination problem, and the trade-off between sortable, random and sequential belongs there.