Microservices
Microservices split an application into independently deployed services that each own their data and talk over the network — buying team autonomy, independent releases and per-service scaling at the price of partial failure, lost transactions, contract versioning and an observability stack; adopt them for a measured organisational or scaling problem, or you build a distributed monolith.
Many teams cannot ship, scale or fail independently inside one deployable: a release train the size of the company, one scaling curve for every capability, one failure domain. Microservices give each team a service with its own code, database, pipeline and runtime, so those become independent — at the cost of turning function calls into distributed-systems problems.
What it is, and what it costs at once
A microservice architecture is a set of small, independently deployable services, each owning a business capability and its own data, communicating through APIs or messages. A request from a client enters through an API Gateway and is routed to the User, Order or Payment service; each has its own database that no other service reads. The Order service that needs to charge a card calls the Payment service over the network; the Payment service calls the external provider; nobody shares a transaction.
The benefits are exactly the ones a monolith cannot give: independent deployment (the payments team releases at 11 a.m. without a company-wide train), independent scaling (20 catalogue instances, 2 payment instances), technology freedom per service, and failure isolation — if properly built, a crashed Notification service does not stop checkout. The costs arrive on day one, not at scale: every arrow in the diagram is a network call that can be slow, fail, or fail partially; every piece of data lives in exactly one place and cannot be joined with data elsewhere; every contract between services must be versioned; and nothing about a request is visible without tracing. The thesis of this lesson is that microservices introduce distributed-systems complexity and must solve a real organisational or scaling problem that is larger than that complexity.
Service boundaries and database ownership
A service boundary is a business capability that one team can own end to end: users, orders, payments, inventory. Boundaries drawn around technical layers (a "database service", a "validation service") or around nouns too small to own anything (a "product name service") produce chatty call chains where every request crosses five services to do one thing. Conway’s law runs in both directions here: the services you can sustain are the ones that match teams that exist, and the organisation will reshape itself around whatever boundaries you draw. If you cannot name the team that owns a proposed service, it is not a service yet.
Database per service is not optional. A service that shares a database with another is coupled at the schema — a migration in one breaks the other, and the "independent deployment" is fiction. Ownership means the Order DB is reachable only through the Order service’s API or events. The consequence is that queries which used to be one JOIN — "orders in the last week with the buyer’s name" — become two calls, or a read model the Order service maintains by subscribing to UserRenamed events. That duplication is deliberate: the Order service stores the buyer’s name at order time because that is the name on the invoice, and because a service that must call another to render its own data is not independent.
- Boundaries follow business capabilities and teams, not layers or entities.
- One database per service; other services reach data only through the owner’s API or events.
- Cross-service reads become API calls or locally maintained read models; accept the duplication.
- A boundary you cannot assign to a team is a boundary you cannot operate.
Network calls and partial failure
Inside a monolith, payments.charge(order) is a function call: it either returns or throws, and it takes nanoseconds. Across services it is an HTTP call that takes 1–5 ms on a good day, and can return a timeout after 3 s having actually succeeded on the far side. That is the defining problem of the architecture: partial failure. The Order service does not know whether the charge happened. Retry, and you may charge twice — the challenge double-charge-after-timeout is this incident. Give up, and you may have a paid order marked failed. The only honest answer is at-least-once calls plus idempotency: the Order service sends an idempotency key with every charge, and the Payment service returns the stored result for a repeated key.
Latency and failure also compound along chains. A request that touches gateway → Order → User → Payment → provider inherits the p99 of each hop, and a single slow dependency with no timeout can hold every thread in the Order service until it, too, is down. Retries make this worse, not better: 3 retries at each of 3 layers is 27 attempts against a provider that is already struggling (Circuit Breaker, retry-storm-took-down-payments). Every call needs a timeout shorter than its caller’s, a bounded retry budget, a breaker per dependency, and a decision about what to do when the dependency is down — fail fast, degrade, or queue for later (Reliability Patterns, Request/Response vs Event-Driven).
Gateway ──▶ Order service budget 2000 ms Order ──▶ User service 35 ms ok Order ──▶ Payment service .... timeout at 3000 ms ◀ longer than the 2000 ms budget Payment ──▶ Provider 2800 ms ok (charge succeeded!) Order ◀── 504 to gateway; order left 'pending'; client retries → second charge? Fix: timeouts nest (provider 1500 < payment 1800 < order 2000), idempotency key on POST /charges
Deployment, versioning and observability
Independent deployment only works if services never require a simultaneous deploy. That makes every API and event schema a versioned contract: changes are additive (new optional fields), never renames or removals until every consumer has moved, following expand-then-contract; consumer-driven contract tests catch a producer change that would break a consumer before it ships. The moment two services must deploy together, you have lost the property you paid for.
Deployment complexity is a platform: a CI pipeline per service, container images, an orchestrator, Service Discovery so that Order can find the current Payment instances, secrets and inter-service authentication, and a gateway. Observability stops being optional: a request that was one log line is now scattered across five processes, so every hop propagates a trace id and every log line carries it (Logs, Metrics and Traces, Distributed Tracing). Without that, "checkout is slow" cannot even be localised — and the challenge the-140ms-database-span shows an N+1 that was invisible until a trace revealed 38 sequential calls across a boundary.
- Contracts: additive changes, expand-then-contract, consumer-driven contract tests; never a coordinated deploy.
- Platform: CI per service, containers and an orchestrator, discovery, secrets, service-to-service auth, a gateway.
- Observability: trace-id propagation on every hop, structured logs with the trace id, per-service RED metrics.
- On-call: each service needs an owner who can be paged; a service nobody owns is an outage waiting to be discovered.
Data consistency
There is no transaction across services. Placing an order and charging for it are two writes in two databases, and one can succeed while the other fails or hangs. The pattern that replaces the transaction is the Saga Pattern: the order is written as pending, the charge is requested, and the order becomes paid or is cancelled by a compensating action; the Distributed Transactions lesson covers why two-phase commit is not the answer. State changes and the events that announce them are written atomically with an outbox table, so a crash between commit and publish cannot lose the OrderPaid event.
The user-visible consequence is eventual consistency: the order list may show pending for a second after the card was charged, the email may arrive before the status flips, and a read model built from events may lag (Event-Driven Architecture). Designing the UI around those pending states is part of the architecture, not a detail — the challenge read-model-shows-stale-order is what happens when it is skipped.
The distributed monolith
The commonest failure is not choosing microservices wrongly; it is building them without the properties that justify them. A distributed monolith has nine services that deploy together because their contracts break otherwise, a shared database that every service reads and writes, and synchronous call chains so deep that any one service being down takes the whole system down. It pays every cost — network latency, partial failure, no transactions, tracing, a platform team — and gets none of the benefits: no independent deploys, no isolation, no independent scaling. The challenge distributed-monolith is the investigation of exactly this system.
The checklist is short. Can each service be deployed alone, today, without coordinating with anyone? Does each service own its data, with no other service touching its tables? Can the system serve *something* when any one service is down? If the answer to any is no, the architecture is a monolith with a network in the middle, and the honest fix is usually to merge back into a Modular Monolith and extract only the one or two services that had a measured reason to exist. Interviewers ask when-microservices to hear the thesis stated plainly: microservices are justified by team autonomy, deployment independence, differing scaling needs and clear domain boundaries — and they cost operational maturity you must already have.
| Property | Microservices | Distributed monolith |
|---|---|---|
| Deploy one service alone | Yes, any time; contracts are backward compatible | No; releases are coordinated across services |
| Data ownership | One database per service | Shared database; migrations break neighbours |
| Behaviour when one service is down | Degraded: timeouts, breakers, queued work | Everything fails: deep synchronous chains |
| Scaling | Per service, by its own load | Whole graph, because every request touches every service |
| What you paid for | Autonomy and isolation | Network costs with none of the autonomy |
Key points
- Independently deployed services, each owning a business capability and its own database, talking over the network.
- The costs are immediate: partial failure on every call, no cross-service transactions, versioned contracts, and mandatory tracing.
- At-least-once calls plus idempotency keys is the honest form of "exactly-once"; nested timeouts and bounded retries keep one slow dependency from taking everything.
- Consistency across services is a saga with an outbox and pending states, not a transaction; design the UI for eventual consistency.
- Justify the split with a measured organisational or scaling problem; a shared database or coordinated deploys means you have built a distributed monolith.
One request across three services, with failures
Monolith vs modular monolith vs microservices
Rebuild and redeploy the whole application; every module restarts; the full regression suite gates the release. ~20 min pipeline, one release train.
Same artifact redeploys, but the change is confined to the payments module and its own tests gate the merge. Faster to review, same blast radius on rollout.
Only the Payment service redeploys, ~3 min, nothing else restarts. Consumers are unaffected as long as the API contract holds.
How data moves through it
One request or event, hop by hop.
- 1Client → API gateway: TLS, JWT verification, rate limit, route
POST /ordersto the Order service. - 2Order service → User service:
GET /users/{id}with a 500 ms timeout to validate the buyer; result cached locally for a minute. - 3Order service → Order DB: insert the order as
pendingplus an outbox row, in one local transaction. - 4Order service → Payment service:
POST /chargeswithIdempotency-Key: order-{id}and a 1.8 s timeout; Payment → provider with 1.5 s. - 5Payment service → Payment DB → Event bus: store the charge, publish
PaymentSucceeded; Order service consumes it and marks the orderpaid. - 6Event bus → Notification / Analytics services:
OrderPaidfans out asynchronously; duplicates are dropped by a processed-event table.
When to use — and when not
- Several teams measurably blocked by a shared release train (lead time in days, cross-team rollbacks) and able to own a service each, including on-call.
- Capabilities with genuinely different scaling or runtime needs — a GPU inference path, a PCI-scoped payment path — that cannot share a process efficiently.
- Domain boundaries that are already stable and proven, ideally as modules in a modular monolith, so extraction follows an existing seam.
- An organisation with the platform maturity to run it: CI per service, orchestration, discovery, tracing, and someone paged per service.
- A team small enough to share one codebase and one release; the network buys them nothing and costs them transactions.
- Domain boundaries still being discovered — a wrong service boundary is far more expensive to move than a wrong module boundary.
- Without observability and a deployment platform in place; services that cannot be traced or deployed independently are a distributed monolith on day one.
- To "scale better" in the abstract: a monolith behind a load balancer with a well-scaled database serves tens of thousands of requests per second.
Tradeoffs
The highest complexity and operational cost of any shape, and the weakest consistency because every cross-service write is eventual. Scalability is the payoff — per service, per team, per runtime — and the only reason to pay.
How it fails
- Partial failure and double charge: a 3 s timeout on
POST /chargesfires after the provider succeeded; a retry without an idempotency key charges twice. - Retry amplification: 3 retries at 3 layers become 27 attempts against a struggling payment provider and finish it off.
- Cascading latency: a slow User service with no timeout holds every Order service thread until Order is down too; no bulkhead, no breaker.
- Shared database or coordinated deploys: a schema migration in Payment breaks Order; releases need a war room — a distributed monolith.
- Lost events: a crash between committing the order and publishing
OrderPaidleaves an order paid with no email and no fulfilment, because there was no outbox.
How it scales
- Per service: the catalogue runs 20 instances and payments 2, each sized to its own load and its own database.
- Per database: each service’s store scales on its own ladder (replicas, cache, partitioning) without touching the others — see Scaling from One User to Millions.
- The gateway, discovery and the broker become the shared infrastructure that must scale for everyone and must be highly available (API Gateway, Service Discovery).
- Fan-out is the limit: a request that touches ten services has ten p99s in series, so deep synchronous chains are replaced with events where the caller does not need the answer now.
How it interacts with databases, queues, caches, APIs and external systems
- Database: one per service; no shared schemas, no cross-database joins; cross-service data is fetched by API or replicated from events.
- Queue / event bus: the primary integration between services for anything the caller does not need synchronously; at-least-once delivery, so consumers are idempotent.
- Cache: per-service caches of other services’ data (the buyer’s name for a minute) to cut chattiness; never a shared cache used as a back channel.
- APIs: REST or gRPC contracts between services, versioned additively; the gateway exposes the public surface and keeps business logic out (API Architecture: REST, GraphQL, RPC, gRPC, WebSockets, Webhooks).
- External systems: each provider integration lives in exactly one service, with a timeout, a breaker and a webhook endpoint for asynchronous confirmations.