Microservices
Independent deployment and independent failure, bought with every cost that arrives when a function call becomes a network call.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
What do you actually get for splitting into services, and what arrives with it whether you wanted it or not?
The system is large, several teams work on it, and someone has proposed splitting it into services. The decision has to be made on grounds better than "large systems are made of services".
Split along the obvious entities — orders, users, payments, catalog — give each an HTTP API and a team, and let them deploy independently. Each service is small and easy to understand.
A function call became a network call, so every call site now has five outcomes instead of two: success, failure, timeout, slow, and succeeded-but-you-did-not-hear. Partial failure is not an edge case; it is the normal operating condition of a distributed system (Failure Propagation).
- A function call became a network call, so every call site now has five outcomes instead of two: success, failure, timeout, slow, and succeeded-but-you-did-not-hear. Partial failure is not an edge case; it is the normal operating condition of a distributed system (Failure Propagation).
- The transaction is gone. "Create order and reserve stock" spanned two tables and now spans two services, so it is a saga with compensations, or an outbox with a consistency window, and either way it is code you now maintain forever (The Dual Write Problem, The Transactional Outbox).
- Every inter-service call is a versioned contract. Two versions are live during every deploy, so every change is expand-then-contract, and a field cannot simply be renamed again (Expand and Contract Migrations, Running Two API Versions in One Service).
- A stack trace stopped covering a request. Debugging now requires distributed tracing that somebody has to build, propagate and operate before the first incident, not during it (Tracing From the Backend's Side, Correlation Ids That Survive Every Hop).
- Operations multiplied: each service needs a pipeline, alerts, dashboards, secrets, an owner, a runtime upgrade path and an on-call rotation. Twelve services is twelve of everything, and that cost is charged monthly.
- Split on entities rather than on behaviour and you get a distributed monolith: services that must deploy together, one user action fanning across six of them, and all of the above costs with none of the independence.
What is actually happening
- Exactly four things change when a call crosses a process boundary, and everything else follows from them.
- 1. Partial failure. The callee may have done the work and failed to tell you. There is no way to distinguish "not done" from "done, response lost", so every write across the boundary needs idempotency and every caller needs a timeout policy (Idempotency in Backends, Timeouts).
- 2. No shared transaction. Atomicity stops at the process. Cross-service consistency becomes a design problem with named patterns and real consistency windows (Eventual Consistency in Practice).
- 3. Contracts and version skew. Types became a wire format. Producer and consumer versions differ during every deploy and sometimes for months (Three Models, Not One).
- 4. Observability across processes. Causality has to be reconstructed from context you propagate deliberately. Without it, "why was this request slow" has no method (Request Context Propagation).
- None of the four is a reason not to do it. They are the invoice, and the point is to know the price before agreeing to it.
- What you buy is precise: independent deployment and independent failure and scaling. If a proposed split does not deliver at least one of those for a component that needs it, it delivers nothing and still charges the full price.
The moment the call crosses a process
Everything in this lesson comes from one substitution. The left-hand call has two outcomes and takes microseconds. The right-hand call has five outcomes, takes milliseconds, can be slow without failing, and can succeed without you knowing.
The fifth outcome is the one that reshapes the design. "It succeeded and the response was lost" is indistinguishable from "it failed", so the caller cannot know whether to retry, which is why idempotency stops being a nicety and becomes a precondition for the architecture.
// two outcomes: returns, or throws const reservation = inventory.reserve(orderId, lines) // and it is inside the same transaction as the order insert
// five outcomes, and a design decision for each
const reservation = await inventoryClient.reserve({
idempotencyKey: orderId, // (a) it may have succeeded already
lines,
}, {
timeoutMs: 800, // (b) slow is a distinct outcome from failed
retries: 2, // (c) only safe because of (a)
onOpenCircuit: 'reject', // (d) decided in advance, not during the incident
})
// and the order insert cannot be in a transaction with itEverything on the right is compensation for losing the guarantees on the left. The idempotency key exists because a timeout does not tell you whether the work happened; the circuit policy exists because the dependency will be down and someone has to have decided what checkout does then. This is not overhead you can skip — it is the actual cost of the boundary.
Forces that justify a service, and reasons that do not
The useful discipline is to require a force, in writing, per service. Each of the first five below is specific and checkable, and each justifies extracting one component — not restructuring the system.
The rejected reasons are not strawmen. They are the ones that appear in real design documents, and each is a plausible-sounding proxy for something that has not been measured.
Which force requires independent deployment or independent failure for this specific component?
when Several teams block on one pipeline and you can state the delay in hours per week.
cost A contract to version forever, plus the full platform baseline. Try flags and trunk-based deploys first (Feature Flags: Rollout, Kill Switches and Debt).
when GPU, 30 GB of memory, or a workload whose scaling curve is nothing like the rest.
cost One service, one boundary, one set of distributed calls. Usually the cheapest justified split.
when Compliance scope, tenant isolation, or a critical path that must survive the rest of the system failing.
cost Duplicate infrastructure and a strict boundary you may not relax later.
when The work genuinely requires a runtime the main deployable cannot host.
cost A second toolchain, second dependency tree, second on-call skill set.
when Checkout must keep working when reporting is broken, and bulkheads inside one process are not enough.
cost Real isolation, real distributed-call costs on that path.
when Never. Size is a symptom; find which of the above it is a symptom of.
cost Every cost in this lesson, no benefit. §119 names this specifically.
when Only if that ownership is currently blocked by deploy contention.
cost Org structure mapped onto deployment topology, which then has to be remapped at the next reorganisation.
when Never. Entity-shaped splits produce distributed monoliths.
cost Chatty synchronous fan-out for a single user action.
What a request looks like afterwards
Draw one user action across the new topology before committing. The question is not whether the diagram is drawable; it is how many synchronous hops sit between the click and the response, because availability and latency both compound along that path.
The version on the right is what a considered split looks like: one synchronous call to the thing checkout genuinely cannot proceed without, everything else asynchronous. The version on the left is what an entity-shaped split produces, and it has a worse availability figure than the monolith it replaced.
How to build it
Most important first.
- Require a named force per service, written down: a team boundary causing measurable deploy contention, a divergent resource profile, a compliance isolation requirement, a forced runtime, or blast-radius isolation for a critical path. "It is a different noun" is not a force.
- Extract one service at a time, along a boundary that is already real in the code — which is what The Modular Monolith gives you. Extracting from an entangled codebase means discovering the boundary and building the distribution at the same time, and both will be wrong.
- Split by capability and change cadence, not by entity. If two services always change together, they are one service that is paying for two.
- Each service owns its data. No shared database, no cross-service joins. A shared schema means you have distributed the code and not the coupling, which is the worst of both (The Repository Layer).
- Build the operational baseline before the second service: tracing with propagated context, correlation ids, structured logs, per-service SLOs, a service catalogue with owners. After the fifth service this work is archaeology (Tracing From the Backend's Side, Structured Logging).
- Make every cross-service write idempotent and every caller bounded: timeout, limited retries with jitter, a concurrency cap, and a defined behaviour when the dependency is down (Retries, Backoff and Jitter, Circuit Breakers, Bulkheads).
- Prefer asynchronous messaging where the caller does not need the answer to finish its own job — it decouples availability instead of multiplying it (Synchronous vs Asynchronous Communication).
- Count the operational cost honestly and staff it. Microservices are a bet that the platform investment is cheaper than the coordination cost of the monolith. That is sometimes true, and it is a calculation, not a principle.
What can go wrong
- The distributed monolith: services that cannot deploy independently because their contracts change together. Full cost, no benefit, and it is the most common outcome.
- A shared database between services, which means an internal change in one breaks another with no contract anywhere.
- A synchronous call chain several services deep, where availability multiplies down the chain and one slow leaf becomes a system-wide latency floor (Failure Propagation).
- Retries at every hop with no budget, so one slow dependency receives exponentially amplified load and cannot recover (Cascading Failure, Retry Storms).
- Nobody owns the end-to-end user journey, so a broken checkout is six teams each confirming their service is healthy.
- Local development requiring a machine to run twelve containers, after which people stop testing integrations locally and find out in staging.
- A service extracted for a reason that later disappears — the team was reorganised — and nobody merges it back, because merging is not a project anyone funds.
- The same request retried after a timeout runs concurrently with the original in the callee. Every cross-service write must be idempotent (Idempotency Keys).
- Two services updating related state concurrently have no shared lock and no shared transaction. Ordering must be established explicitly — a version, a sequence, or a single owner for the write (Optimistic Concurrency).
- Events consumed out of order, or consumed before the producing transaction commits, are the standard event-driven race (Writing Event Consumers, The Transactional Outbox).
- The internal network is not a trust boundary. Every service authenticates its callers and authorizes per object, or the first SSRF or compromised pod reaches everything (The Trust Boundary, SSRF — When the Backend Fetches a URL).
- Attack surface multiplies: more endpoints, more credentials, more dependency trees to patch, more images to rebuild (Dependency Security).
- The genuine security gain is containment: a compromise of one service reaches only what that service's identity can reach, if and only if per-service credentials and network policy actually exist (Defence in Depth).
- Authorization consistency becomes a real problem — the rule must hold across services, which usually means a shared library or a policy service, and both need their own failure-mode design (Authorization in Backends).
- "The system is big, so we need microservices." §119 exists for this sentence. Size is not a criterion. Deploy contention, resource-profile divergence, isolation requirements and forced runtimes are.
- "Microservices scale better." Stateless services scale horizontally whether there is one of them or twenty. What scales independently is each component, which matters only when their profiles differ (Horizontal vs Vertical Scaling).
- "Small services are simpler." Each service is simpler. The system is substantially more complex, and the system is what your users experience.
- "We will split now to avoid a painful migration later." Splitting before the boundaries are understood produces wrong boundaries, and a wrong service boundary is far more expensive to move than a wrong module boundary.
- "Each team owns a service, so we are decoupled." Only if the services can deploy independently. Measure it before believing it.
- "Serverless functions instead of services avoids this." The unit changed; every distributed cost in this lesson stayed (Serverless Backends, Comparing Backend Architectures).
Operating it
- Distributed tracing with propagated context is not optional. Without it there is no method for finding where a request spent its time, only guesswork per team (Tracing From the Backend's Side).
- Per-service SLOs plus a dependency map. When a downstream breaches, the upstream should be able to say so before its users do (Availability, SLOs and Error Budgets in Software Architecture).
- Track deploy independence directly: how many services had to ship together for the last twenty changes. A rising number is the distributed monolith forming, and it is the single most useful metric in this module.
- Track cross-service call fan-out per user request. A checkout that touches nine services synchronously has a latency and availability profile nobody chose (Fan-Out: Waiting for the Slowest of Seven in Observability & Performance).
- At 10x traffic a monolith usually just needs more instances. Services help here only when one component's resource profile genuinely diverges from the rest — which is a specific, checkable claim, not a general one.
- At 10x team size services help materially, because deploy contention and code ownership are the costs that grow with headcount and services address exactly those.
- At 100x, both the benefits and the failure modes compound: the platform investment pays back across hundreds of services, and a synchronous call graph that deep will produce cascading failures unless load shedding, budgets and bulkheads are in place from the start (Cascading Failure).
- The reverse is also legitimate and under-practised: consolidating services back when the force that justified them is gone.
- You trade in-process correctness guarantees for deployment and failure independence. That is the entire trade, and it is worth it only when the independence is something you need.
- Latency increases: serialization, network, deserialization on every hop, on the request path, every time (What Serialization Costs).
- Availability multiplies down a synchronous chain. Three dependencies at 99.9% each give you worse than 99.9%, and nobody notices until the arithmetic is done (Synchronous vs Asynchronous Communication).
- Operational cost is per service and permanent. Twelve services means twelve pipelines, twelve alert sets, twelve dependency trees to patch (Scoring Operational Complexity in Cloud & Infrastructure).
- Testing gets harder in a specific way: unit tests get easier, and confidence that the system works end to end gets much more expensive (Contract Tests Between Services).
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- SCALE-SPECIFICBelow roughly three teams the costs listed here dominate the benefits in essentially every case — there is no deploy contention to relieve. The break-even is a function of team count and the operational capacity you can staff, not of request rate or codebase size.
- GENERALThe four consequences of crossing a process boundary — partial failure, no shared transaction, version skew, cross-process observability — hold for every technology, protocol and platform. gRPC, a service mesh and a managed platform change the ergonomics and remove none of them.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Distributed Systems — consensus, ordering, partition behaviour and the impossibility results that sit under the patterns named here.
- — System Design — the interview version of this decision, where naming the force beats naming the topology.