What Changes When Work Crosses a Machine
An in-process function call costs nanoseconds and either returns or throws. The same call across a network costs milliseconds, serialises both ways, waits in three queues you cannot see, and has a third outcome: no answer at all.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The four taxes on a call that leaves the process
Crossing a machine boundary adds four costs, and only one of them appears in the callee's latency metric. There is the network itself — round-trip time, which is a floor set by distance. There is serialisation and deserialisation on both ends, which is CPU proportional to payload size. There is queueing: the client's connection pool, the server's accept queue, the thread pool waiting for a worker. And there is a new failure mode — the call that neither succeeds nor fails but simply does not answer, which the caller must resolve with a timeout it has to choose.
The reason the arithmetic never adds up is that the callee measures only its handler. If service A reports 12ms and service B reports 8ms and the operation takes 180ms, the missing 160ms is in the gaps: connection acquisition, serialisation, network, queueing at the far end, and deserialisation on the way back. Those gaps are exactly what a distributed trace makes visible and what per-service dashboards structurally cannot show.
None of this is an argument against distributing a system — there are excellent reasons to, and they are the subject of Software Architecture rather than this domain. It is an argument for counting the boundaries a request crosses, because each one is a fixed tax that gets paid on every single request forever.
| Cost | In-process | Across a network boundary | Visible in the callee's metrics? |
|---|---|---|---|
| Invocation | Nanoseconds — a stack frame | Round-trip time: sub-millisecond same-rack to hundreds of milliseconds cross-region | No |
| Arguments | A pointer | Serialise, transmit, deserialise — CPU on both ends, proportional to payload | Partially |
| Waiting | None | Connection pool, accept queue, worker pool — three queues, each able to saturate | No |
| Failure modes | Returns or throws | Returns, throws, or never answers — the caller must invent a timeout | No |
| Observability | A stack trace | A distributed trace, if context propagated; otherwise nothing | N/A |
Where the missing milliseconds are
Take a concrete accounting. Service A calls service B; B reports p50 handler time of 8ms; the caller observes 47ms. The 39ms difference is not mysterious once the hops are named, and each named hop has a different owner and a different fix.
Connection acquisition is the one most often forgotten. If the client pool is exhausted, the call waits for a connection before a single byte moves — and that wait is attributed to "the network" or "the downstream service" by everyone who has not looked. It is neither: it is a local queueing problem with a local fix, and it behaves exactly like the pool saturation described in Connection Pool Saturation: Waiting in Front of an Idle Database.
The other reliably invisible cost is the server-side accept and worker queue. The handler histogram usually starts when a worker picks the request up, not when the request arrived. Under load, the gap between arrival and pickup can dwarf the handler time, which is why a service can report excellent latency while its callers time out.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Callee handler p50 | 8ms | What the downstream team sees on their dashboard, and it is accurate | normal |
| Connection acquisition | 14ms | Client pool exhausted — a local queue, not a network problem | smoking gun |
| Serialise + deserialise (both ends) | 9ms | Proportional to payload; the 400KB response is doing real work here | suspect |
| Network RTT | 2ms | Same availability zone; genuinely cheap | normal |
| Server accept-to-pickup | 11ms | Request waited for a worker before the handler histogram started | smoking gun |
| Caller-observed total | 47ms | 39ms of it is invisible on the callee's dashboard | suspect |
Counting boundaries
Because each boundary is a fixed per-request tax, the number of boundaries a request crosses is a first-class design number. A request that touches six services pays the tax six times, and if those calls are sequential the taxes add rather than overlap — which is the subject of Sequential or Parallel: Same Work, Different Latency.
The tax also compounds with depth. A gateway calling a service that calls two services that each call the database is four levels deep; a timeout at the top must be larger than the sum of everything below it, or the outer call gives up while inner work is still running and burning capacity. Getting that hierarchy wrong produces the pattern where load increases, timeouts fire, retries pile on, and the system does more work the worse it gets (see Retry Storms: The Load You Generated Yourself).
The diagnostic habit worth building: before optimising any individual service, draw the request's path and count the crossings. Frequently the highest-leverage change is removing a hop — collapsing two services that always call each other, or batching what was a per-item call — rather than making any single service faster.
Key points
- A network boundary adds round-trip time, serialisation on both ends, three queues, and a third outcome — no answer at all.
- A callee's latency histogram measures handler time only; connection acquisition and accept-queue wait are invisible to it and often larger.
- Caller-observed latency minus callee-reported latency is a diagnostic quantity worth graphing on its own.
- The number of boundaries a request crosses is a design number: each one is a fixed tax paid on every request forever.
- Removing a hop often beats optimising any single service on the path.
Progressive depth
Overview
Crossing a machine boundary adds network time, serialisation, queueing and a new failure mode. Each boundary is a tax paid on every request.
Practical
Graph caller-observed latency against callee-reported latency for every dependency. The gap is the tax, and it is where connection pool waits and accept-queue waits hide.
Advanced
Count boundaries and depth per request path. Sequential hops add; parallel hops take the maximum plus tail risk. Timeout budgets must decrease with depth or outer timeouts fire while inner work continues.
Internals
Each crossing traverses a client connection pool, a socket send buffer, the network, a server accept queue, a worker pool, and the reverse path back. Any of those queues can saturate independently, and only the handler is instrumented by default.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Client → pool: all 20 connections are in use, so the call waits 14ms before any byte is sent.
- 2Client → network: the request is serialised and crosses the network in 2ms — genuinely the cheapest hop in the chain.
- 3Server → accept queue: the request sits 11ms waiting for a free worker, before the handler timer starts.
- 4Handler → callee dashboard: 8ms of actual work is recorded and reported as excellent latency.
- 5Response → client: a 400KB payload is serialised, transmitted and deserialised, and the caller records 47ms for an operation the callee is certain took 8ms.
- • "The downstream service says it is fast, so the problem is our code." Their metric measures handler time; the tax is in the gaps around it.
- • "It is a network problem." Round-trip time within an availability zone is usually the smallest component; check pools and queues first.
- • "Adding a service is free, it is just one more call." It is a permanent per-request tax paid on latency, CPU and failure surface.
- • "The sum of service latencies is the request latency." It is a lower bound, and usually a distant one.
- • "Serialisation is negligible." For large payloads it is measurable CPU on both ends, and it scales with the payload the API chose to return (see Payload Size: 20KB, 200KB, 5MB).
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Caller-observed latency per dependency alongside that dependency's own reported handler latency — the difference is the boundary tax.
- • Connection pool wait time and pool utilisation on the client side, as a distinct metric from request duration.
- • Server-side accept-to-handler-start time, so queueing before pickup is separated from handler work.
- • Serialised payload size per call, since serialisation CPU tracks it directly.
- • Hop count and depth per request path, from a distributed trace rather than from an architecture diagram nobody updated.
- • Remove boundaries where two services always call each other on the same path — the tax disappears entirely rather than getting smaller.
- • Fix client-side pool saturation and server-side queueing before touching handler code; they are frequently the larger share.
- • Shrink payloads so serialisation cost falls on both ends, using field selection or purpose-built response models.
- • Convert per-item calls into batch calls so N boundary crossings become one.
- • Set timeout budgets that decrease down the call hierarchy, so an outer timeout cannot fire while inner work continues (see [[timeouts-and-latency]]).
- • Caller-observed p99 per dependency, compared against the same window before the change — this is the number the user experiences, not the callee's histogram.
- • The caller-minus-callee gap specifically, which should shrink if the fix targeted queueing or serialisation.
- • Hop count per request path from traces, confirming a removed boundary actually left the path rather than moving.
- • Downstream service CPU, to confirm a batching change reduced total work rather than concentrating it.
- • Collapsing services to remove a hop trades latency for the coupling and deployment independence that motivated splitting them.
- • Batching reduces boundary crossings and increases payload size, latency for the first item, and blast radius on partial failure.
- • Larger connection pools reduce client-side queueing and increase memory and file descriptor use, and can push the queue onto the server.
- • Tight timeout hierarchies improve failure behaviour and make the system less tolerant of legitimate slow paths.
- • An alert on the caller-observed-minus-callee-reported gap per dependency, which catches pool and queue regressions that no single service dashboard sees.
- • A trace-derived check on hop count for critical request paths, failing when a new boundary appears on a hot path.
- • Connection pool saturation alerts on the client side, independent of latency alerts.
- • A review rule that a new synchronous service call on a critical path states its expected added latency and its timeout.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe 47ms-versus-8ms breakdown is invented to show where the missing time hides. Real proportions depend on payload size, pool configuration, network topology and load.
- ENVIRONMENT-SPECIFICRound-trip time within a rack, across availability zones and across regions differ by orders of magnitude; "the network is cheap" is only true at the smallest of those scales.