A Remote Call Is Not a Function Call

The question this answers

The question

My RPC client makes a remote call look exactly like a local one. What is that hiding?

The guarantee — the property claimed, and its scope

A local call guarantees exactly-once execution with a definite result: return or throw. A remote call guarantees at-most-once *attempt delivery* and nothing about execution — it may execute zero, one or more times, and the caller may not learn which.

Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.

What a node knows — observation versus inference

The caller knows what it sent, when it sent it, and what came back — if anything came back. It does not know whether the callee received it, started it, finished it, or committed it. A returned value is proof the work happened; the absence of a returned value is proof of nothing.

A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.

What guarantee?→What does a node know?→How does it work?→What can fail?→How does it fail?→Where is coordination?→What holds under failure?→How does it recover?→How would you know?→What is the simpler thing?

rpcabstraction leaksemanticsfoundations

The three outcomes, not two

A local function call has a complete outcome space: it returns a value, or it raises. Both are *results* — the callee reached a conclusion and told you. Your error handling is exhaustive because the language guarantees the space is exhaustive.

A remote call has three outcomes: a response, an error response, and no response. The third is not a variant of the second. An error response means the callee decided something and told you; no response means you have no information at all, and the work may be finished, in progress, or never started. This is the whole of A Timeout Tells You Nothing About Whether It Happened, and it is the reason a catch block around a remote call is usually wrong: it merges a definite negative with a total unknown.

This is why the interesting question is never "how fast is a remote call" but "what may I conclude from what I observed?" The numbers vary by deployment; the conclusions do not.

	Local call	Remote call
Returned a valueprotocol	Executed exactly once	Executed at least once — a retried duplicate may also have executed
Threw / erroredtypical	Did not complete; state unchanged past the throw point	Callee reached some decision. Partial effects may persist. A proxy may have synthesised the error without the callee seeing it
No answerprotocol	Cannot happen — the call is on your stack	No information. Zero, one or more executions are all consistent with this
Latencyassumption	Bounded by your own scheduling	Unbounded: a queue, a retry, a GC pause or a route flap is indistinguishable from work
Callee availabilityprotocol	Shared fate — if you run, it runs	Independent — it may be gone while you are fine, or fine while unreachable

What each observation licenses you to conclude

Latency variance matters more than latency

A remote call being slower than a local one is the least interesting difference, because it is the one you can measure and budget for. The difference that changes designs is that its latency has a long, fat tail with no upper bound: a queue somewhere, a retransmit after packet loss, a garbage collection pause on the callee, a connection re-establishment, a DNS lookup that fell out of cache. A local call has variance too, but it is bounded by your own runtime.

The consequence is structural, not numerical. Because the tail is unbounded, you must impose a bound yourself — a deadline — and the moment you impose a deadline you have created the third outcome. The timeout is not a defence against slowness; it is the thing that manufactures ambiguity. You cannot have one without the other, and a system with no timeouts simply converts the tail into permanently-held threads and connections instead.

A second consequence: a caller with a p99 dependency latency of 20ms and a fan-out of ten is not making a 20ms call. Waiting for the slowest of ten samples from a distribution with a fat tail is a different distribution — that is Performance’s subject and worth reading before you design any fan-out.

Independent failure is what the abstraction cannot hide

The historical version of this lesson is about transparency: RPC frameworks that made a network call look like a method invocation, so that engineers wrote code with local semantics and shipped systems with remote failure modes. Modern frameworks are more honest — deadlines are first class, errors carry retryability hints — but the same trap arrives dressed as an SDK, an ORM with lazy relations across a service, or a client library with a built-in retry you did not configure.

The tell is always the same: is there anywhere in the code where a network round trip is invisible at the call site? A property access that triggers a fetch, a for loop that turns into N sequential requests, a constructor that resolves service discovery. Each of these is a place where someone will write logic assuming the local outcome space.

The other half is failure independence. A local callee cannot be down while you are up. A remote one can be down, or up and unreachable, or up and reachable but too slow to matter, or up and serving a version whose contract differs from the one you compiled against. None of those states exist for a function call, and all of them need a behaviour.

1// Local: two outcomes. The catch is exhaustive.
2try {
3  const receipt = charge(order)      // returned => it happened, once
4  markPaid(order, receipt)
5} catch (e) {
6  markFailed(order)                  // threw => it did not happen
7}
8 
9// Remote: three outcomes. The catch is NOT exhaustive —
10// it silently merges "B said no" with "B said nothing".
11try {
12  const receipt = await billing.charge(order)  // returned => at least once
13  markPaid(order, receipt)
14} catch (e) {
15  // e may be: a 4xx decision, a 5xx decision, a synthesised
16  // gateway error, or a deadline that expired while B committed.
17  markFailed(order)                  // <-- wrong for the last case
18}
19 
20// What the remote version actually needs:
21const outcome = await billing.charge(order, { key: order.id })
22switch (outcome.kind) {
23  case 'succeeded': markPaid(order, outcome.receipt); break
24  case 'rejected':  markFailed(order, outcome.reason); break
25  case 'unknown':   markPending(order); break   // resolve later, do not guess
26}

The same three lines, two completely different outcome spaces

What survives the boundary, and what does not

Some things you take for granted locally simply do not cross. Object identity does not: what arrives is a copy, and mutating it changes nothing at the source. Exceptions do not: what arrives is a serialised description of an exception, at best, and typically a status code chosen by whatever spoke last, which may be a proxy. Ordering does not survive independent connections. Transactions do not cross at all — there is no ambient rollback for work another service committed, which is why the transactions module exists and why compensation is not undo.

What does cross, if you deliberately send it, is data: an identifier you chose, a deadline you computed, a version you expect, a correlation id for the trace. Everything a local call gets from the runtime for free, a remote call must carry explicitly in the payload. That is the practical definition of "designing for the boundary".

Identity becomes a copy — pass an id, not a reference to a live object.
Exceptions become status codes — encode retryability rather than hoping the caller guesses.
Ambient context becomes explicit fields — deadline, trace id, tenant, idempotency key.
Ordering becomes a sequence number or a version, or it does not exist.
Atomicity stops at the boundary and does not resume on the other side.

Key points

A local call has two outcomes; a remote call has three, and the third carries no information.
A returned value proves execution happened at least once — never exactly once.
An error response is a decision by someone; it may not be a decision by the callee.
The unbounded latency tail forces a deadline, and the deadline is what manufactures ambiguity.
Anything a local call gets from the runtime for free must be carried explicitly across the boundary.

The chain, answered

Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.

How it works

• The caller serialises arguments — a copy, not a reference — and chooses a deadline.
• The transport establishes or reuses a connection, resolves a name, and may retry beneath your code without telling you.
• The callee deserialises, may queue the request behind others, executes, and commits.
• The callee serialises a response, which may be lost after the work is durable.
• The caller either receives something within the deadline, or receives nothing and must act without knowing which of the previous four steps completed.

What can fail at the boundary

• Name resolution fails or returns a stale address for a host that has been replaced.
• The connection is refused, reset mid-request, or silently blackholed by a middlebox.
• The request is queued behind slow work at the callee and executes after the caller has given up.
• The response is generated and lost, so the effect is durable and invisible.
• A retry inside the client library duplicates the request without the application code being aware there was one.

How it fails — what an operator sees

• Merged error handling: timeouts are counted as failures and the operation is marked failed while the callee committed. The operator sees a stuck record and no error in the callee’s logs — because there was none.
• Hidden N+1 across the boundary: a loop that reads like local property access issues one request per item. The operator sees a request-count graph on the dependency that tracks page size, and a p99 that grows linearly with result set size.
• Thread or connection exhaustion from an unbounded wait: with no deadline, one slow dependency parks every worker. The operator sees the caller at low CPU, high latency, and a saturated pool while the callee looks merely slow.
• Silent library retry: the SDK retries a non-idempotent POST twice on a connection reset. The operator sees duplicate effects with a single application-level log line and no retry metric, because the retry happened below the layer that logs.

Where coordination is required

• A single request/response needs no coordination — but it also provides no agreement. Both sides can end the exchange with different beliefs about what happened.
• Making the outcome unambiguous requires shared state: a caller-chosen identifier the callee records, so a repeat is recognisable. That record is a coordination point with its own availability.
• Making a *sequence* of remote calls atomic requires far more — see Atomicity Stops at the Process Boundary, and expect the answer to be "you cannot; you compensate".

What still holds under failure

• Whatever the callee committed stays committed regardless of whether the caller heard about it.
• The caller’s view and the callee’s view of the same operation diverge, and stay diverged until something reconciles them.
• In-flight requests that outlive their deadline continue to execute; nothing about the caller giving up reaches the callee unless cancellation was explicitly propagated.

How it recovers

• Detect: separate the timeout counter from the error counter at the call site, so the ambiguous outcomes are visible as their own class.
• Contain: give every remote call a deadline derived from the caller’s remaining budget, so one slow dependency cannot consume the whole request.
• Recover: retry only what is safe to repeat, carrying the same caller-chosen identifier so a duplicate collapses.
• Reconcile: for the unknown outcome, park the work in an explicit pending state and resolve it against the callee later — do not guess.
• Verify: compare caller-side and callee-side records of the same operations; the delta is exactly the population that the third outcome created.

How you would know

• Requests per logical operation on each dependency — a number greater than one means a retry or an N+1 that the call site does not show.
• Outcome distribution split three ways: response, error response, no response. Most dashboards show two.
• Deadline utilisation: the fraction of the caller’s budget each dependency consumes, which is what tells you where to spend the next 50ms.
• Connection churn and pool wait time at the caller, which is where an unbounded remote call first becomes visible as local saturation.

When it helps

• Always — this is a property of the boundary, not an option. The reasoning is most valuable at the first call site a junior engineer writes against a new service.
• Especially where the operation has an external side effect, because the third outcome is expensive exactly there.

When it hurts

• Applying the full ceremony — deadlines, keys, three-way outcome handling — to an internal idempotent read costs code and buys nothing.
• Over-explicit contracts on a boundary you are about to delete: if the two services should be one module, the cheapest fix is removing the boundary, not hardening it.

Simpler alternatives

• Do not make it a remote call: keep the callee in-process behind an interface, so you get local semantics and can extract it later once the boundary is proven.
• Make it asynchronous on purpose: hand the work to a durable queue and return an identifier, converting an ambiguous synchronous outcome into a state machine you can inspect.
• Batch it: one call carrying N items has one ambiguity instead of N, at the cost of coarser partial-failure reporting.
• Use a naturally idempotent shape — send a desired state rather than a delta — so the third outcome stops mattering.

A local call has two outcomes; a remote call has three

A local call has two outcomes. A remote call has three.

Same call site, same catch block. Choose what happened on the wire and read what the caller is entitled to conclude.

What happened?

If this were a local call

try {
  const receipt = charge(order)   // returned => it happened, once
  markPaid(order, receipt)
} catch (e) {
  markFailed(order)               // threw => it did not happen
}

Two outcomes, and the language guarantees the space is exhaustive. Your error handling is complete because it cannot be incomplete.

What actually happened, and what you saw

executions at the callee

effects durable?

yes — committed

caller observed

nothing, until the deadline expired

outcome class

no response — the third outcome

The work is durable. Nobody told the caller. The catch block runs and calls markFailed(order) — which is flatly wrong: the charge is durable and the order is now marked failed.

Caller and callee under: Committed, then crashed before replyingsimplified

delivereddelayed (dashed, long)duplicated (×2)dropped — stops short, never arriveswritecrashdecide

The caller’s lane carries no arrival before the deadline. Everything to the right of that gap is invisible to it — including, in three of these worlds, work that is already durable.

The interesting question is never how fast a remote call is. It is what may I conclude from what I observed. Three of these eight worlds produce identical evidence at the caller, and two of those three left durable effects behind. What crosses the boundary is data you deliberately sent — an id, a deadline, an idempotency key. Everything a local call gets from the runtime for free must be carried explicitly.

protocolThe outcome space is not a property of any framework: it follows from the callee being a separate process reached by messages. The stage names and the timings in the diagram are a simplification of a real RPC path.

What people believe, and what is true

Claim

A good RPC framework makes remote calls behave like local ones.

Reality

It makes them *look* like local ones. The outcome space, the failure independence and the latency tail are properties of the network, not of the API surface.

Claim

If I handle exceptions properly the remote call is safe.

Reality

Exception handling covers the two outcomes a language has. The third arrives as a timeout, and a catch treats it as a definite failure, which is the bug.

Claim

gRPC is fast, so the difference is only microseconds.

Reality

Median latency is not the difference that matters. Independent failure and the unbounded tail are unchanged by an efficient wire format.

Claim

The client library handles retries for me.

Reality

It handles retries *for you*, including on operations that are not safe to repeat, usually without a metric you can see. Find out what it retries before you rely on it.

Go deeper

Only the levels this lesson can honestly fill — a missing level is a claim nobody had.

Overview

Local calls return or throw. Remote calls return, throw, or say nothing — and saying nothing tells you nothing about whether the work happened.

Practical

At every call site: set a deadline from the remaining request budget, carry a caller-chosen id, and branch three ways rather than two. Then find out what your client library retries on your behalf, because that is where the duplicate POSTs come from.

Advanced

This is the argument of Waldo et al., "A Note on Distributed Computing" (1994): the four differences that no abstraction can hide are latency, memory access, partial failure and concurrency. Thirty years of frameworks have improved the ergonomics of all four and eliminated none of them. The design consequence is that boundaries should be chosen for failure semantics first and convenience second — a boundary in the wrong place cannot be fixed by a better client.

Apply it

Build it, then break it

🔧 Take a handler that calls three services in sequence with try/catch and rewrite it so every call branches three ways, with a deadline budget shared across all three.

Interview questions

💬 Your RPC call returned a value. What exactly do you know about how many times the work executed?
💬 Why does adding a timeout to a remote call create a problem you did not have before?
💬 A colleague wraps every remote call in try/catch and marks the operation failed in the catch. What breaks, and when?

← What Actually Makes a System Distributed The Network Changes Everything →

The question this answers

The three outcomes, not two

Latency variance matters more than latency

Independent failure is what the abstraction cannot hide

What survives the boundary, and what does not

Key points

The chain, answered

A local call has two outcomes; a remote call has three

What people believe, and what is true

Go deeper

Apply it

Related