Backend Engineering
How do I design, build, scale, secure, debug and operate the backend of a real production system? Not how to create an endpoint — what happens from the moment a request reaches your service until data is returned, persisted, queued, cached, retried, observed and served reliably at scale.
How do I design, build, scale, secure, debug and operate the backend of a real production system?
Every lesson here starts from a requirement someone actually asked for, shows the implementation that comes to mind first, and then shows how that implementation behaves at 3 a.m. under load with a dependency degrading. The obvious build is obvious for a reason — it is also where production failures come from.
The order is the argument. What the contract promises decides what must be synchronous; what must be synchronous decides what belongs in a transaction; what is in a transaction decides what can safely be retried. Skip a step and you get a design that is locally reasonable and globally wrong.
This is the integration domain
Backend Engineering does not own the depth. It owns the point where the depth meets a running service that someone is paged for.
API Design decides the contract: the resource model, the status codes, the versioning story, what the caller is promised. That is a different question from this one.
This domain builds and runs the service behind that contract — the handler, the transaction, the pool, the queue, the retry, the deploy, the log line you will read during the incident.
Databases, Security, Performance, Cloud, OS & Networking and Computer Architecture each own a layer in full. Lessons here teach the application-side mechanism and hand the rest off by name.
Flagship experiences
The parts of this domain that are not reading.
POST /orders from the client through DNS, TLS, the proxy, the runtime, middleware, the router, authorization, the handler, the transaction, the queue and back. Every hop is a separate participant, and every one of them can fail on its own.
A real /checkout endpoint, step by step — with the hard question asked at each step before the answer is given. Where does the transaction end? What if payment times out? What if the commit succeeds and the publish does not?
Ten rungs, and each one appears only when a problem forces it. Read the problem first: a capability adopted without the problem that justifies it is cost with no benefit.
Inject a fault and read the signals. Several different faults raise p99, so p99 diagnoses nothing — the exercise is finding the signal that tells them apart.
The cross-domain spine: your handler, the driver, the socket, the syscall, the scheduler, the cache hierarchy, the core. You do not need this daily. You need it when the numbers stop making sense.
Where should this work run, what should it run on, and where does the data belong. Every leaf names what it buys and what it costs.
It was 100 ms and it is now 3 s. Work from the symptom to the cause through deploys, queries, pools, dependencies, the event loop and the queue.
Production situations with the cause unlabelled. Each one carries the trap — the wrong fix that looks right and makes the next incident worse.
What each question is really testing, what a strong answer sounds like, and the red flags that separate a remembered rule from a working model.
Learning modules
Twenty-eight modules, from what a backend actually is once the framework is removed to debugging one at 3 a.m.
What a backend actually is once the framework is removed: the request lifecycle end to end, which responsibilities belong to the server because they cannot be trusted to the client, and why every input from outside the process is untrusted.
What a server does between a socket and a response: accept, parse, build a request object, route, execute, serialize, write bytes. The part frameworks hide most completely.
Event loops, threads, workers and processes — the runtime model decides what "slow" means for your service. Taught as concurrency models rather than framework slogans.
How a method and a path become a function call, how precedence resolves ambiguity, and what a handler should and should not be responsible for.
The pipeline every request passes through, why its order is a correctness decision and not a style one, and how cross-cutting concerns compose without leaking into handlers.
Service layers, repositories and the transport/application/domain/infrastructure split — including when each is genuine structure and when it is ceremony that adds indirection without behaviour.
Three different validations that are routinely confused: is this well-formed, is this allowed by the business, and is this consistent with what the database already holds.
Turning runtime objects into bytes and back, what that costs in CPU and allocation, and why the database row is the wrong thing to hand a client.
Establishing who is calling: credentials, sessions, tokens, OAuth and API keys, from the backend's side of the problem rather than the protocol's.
Deciding what the caller may do — role-based, attribute-based, and the object-level check whose absence is the most common serious backend vulnerability there is.
ORM, query builder or raw SQL as an engineering decision with consequences, plus the query patterns and pool limits that decide how a backend behaves under load.
Which operations belong in one atomic unit, why a network call inside a transaction is a resource problem, and what to do when a commit and a message must both happen.
Cache-aside, invalidation, stampedes and the local-versus-distributed decision — including the cases where a cache adds a consistency problem and buys nothing.
Work that does not belong in the request path: deciding what to defer, the queue lifecycle from enqueue to dead-letter, and what happens when producers outrun consumers.
Commands ask for something to happen; events state that it did. What that distinction changes about coupling, naming, consumers and the consistency of everything downstream.
Every call leaving your process can be slow, wrong or absent. Timeouts, retries, backoff, circuit breakers, bulkheads and the rate limits you both enforce and obey.
Inbound HTTP you do not control: signature verification on the raw payload, duplicate delivery as the normal case, and ordering you cannot assume.
The property that makes retries safe. Keys, storage, scope and expiry — and the difference between a queue delivering once and your business logic acting once.
Two requests, one row. Optimistic versioning, pessimistic locks, atomic operations, and the bounded-resource thinking that keeps a burst from becoming an outage.
An error taxonomy that maps causes to responses, boundaries that stop internals leaking, and the logs, metrics and traces that let you answer questions you did not anticipate.
Uploads that do not go through your process, the bucket/key/object primitive underneath every provider's SDK, and what still has to happen after the bytes land.
What belongs in code versus runtime configuration, why secrets are a separate problem, and a test strategy chosen by what each layer can actually prove.
Shipping a running service without dropping requests: containers, graceful shutdown, health checks, rolling deploys and migrations that survive two versions at once.
Statelessness, load balancing, autoscaling signals, pagination, batching and streaming — the specific techniques, and the problem each one is a response to.
The checklist every service owes: injection, SSRF, dependency risk, secrets discipline and defence in depth, from the implementer's side rather than the attacker's.
Monolith, modular monolith, microservices and event-driven, compared honestly — with the distributed-systems costs that arrive the moment a function call becomes a network call.
The API was 100 ms and is now 3 s. Working from symptom to cause through deploys, queries, pools, dependencies, the event loop and the queue.
A model choosing a tool is a client choosing an endpoint. Authorization, budgets, timeouts and audit still belong to the backend, not to the prompt.
Reference
For when you already know roughly what you are looking for.
Nine levels, each unlocking something you can actually build.
The searchable index: what a problem sounds like, and what to think when you hear it.
Ten pairs people genuinely confuse, with the confusion named.
The same primitive across AWS, GCP and Azure — and where the mapping misleads.