Who Runs What, and What Happens When a Worker Goes Quiet

A scheduler does four separable things, and conflating them makes both design and debugging harder. Placement: which worker runs this task, subject to resource requirements, locality preferences and constraints. Admission and capacity: how much work is allowed in at once, and what happens to the rest — which is Decide at the Door Whether the Capacity Exists and Backpressure Is a Signal That Has to Travel — and Reach Someone Who Can Slow Down applied to jobs rather than requests. Liveness tracking: deciding which workers and tasks are still alive. Re-execution: acting on that decision.

The first two are optimisation problems, and getting them wrong costs efficiency. The second two are correctness problems, and getting them wrong costs data. That asymmetry deserves to be visible in how you think about a scheduler: a mediocre packing algorithm wastes money, while a mistaken liveness decision duplicates a payment.

Note also that a scheduler is a coordination point in the sense this domain means. Its availability bounds the job’s ability to start new work; its view of the world is the only global view; and if it holds job state only in memory, its restart is a job-wide event. Whether it holds that state durably is one of the more consequential facts about any scheduler you depend on.

A worker stops heartbeating. The scheduler waits its threshold and then re-runs the task elsewhere. But the worker may be entirely fine — a long garbage-collection pause, a saturated network interface, a partition between it and the scheduler while its connection to the database is perfectly healthy. The original task keeps running. Now two attempts are executing the same work at the same time, and neither knows about the other.

This is Crashed or Just Slow: The Distinction You Cannot Make with a compute framework’s consequences, and it is not avoidable by tuning. Raise the threshold and genuinely dead workers stall the job for longer. Lower it and you duplicate more often. There is no setting that distinguishes the two cases, because no such setting can exist: the information required is not available to the scheduler. The correct response is not to fix the detector but to make the duplicate harmless, exactly as A Timeout Tells You Nothing About Whether It Happened concluded.

Frameworks make it harmless with an exclusive commit: each attempt writes to a private location, and the first to finish atomically claims the destination. The loser discards its work. This is why a task must not have external side effects — the commit convention controls the framework’s own output and nothing else. A task that inserts rows, calls an API, or sends a message has performed that effect regardless of who wins the commit race.

The second thing you need is fencing. If the task holds a lease or a lock, the re-executed attempt must be able to invalidate the original rather than merely coexist with it. Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely is the mechanism: the new attempt carries a higher token, and the downstream system rejects writes carrying an older one. Without fencing, a scheduler that presumes death is arming a second writer against a first one that is still alive.

Placement takes a task’s requirements — cores, memory, an accelerator, a locality preference, an anti-affinity rule that keeps replicas apart — and finds a worker that satisfies them. It is a bin-packing problem, it is NP-hard in general, and every real scheduler uses heuristics: first fit, best fit, scoring functions, a queue per priority class.

What makes it a *distributed* problem rather than an algorithms exercise is that the scheduler’s view of capacity is stale by construction. Reported free memory was true when it was reported. A worker may have started something since. Two schedulers, or two scheduling loops, may both place a task into the same free slot. Systems handle this with optimistic placement plus rejection: the worker refuses what it cannot run, and the task returns to the queue. That is optimistic placement in the ordinary sense — assume it will fit, detect when it does not, rather than trying to hold an accurate global view.

The failure to watch for is resource fragmentation: enough total capacity exists but no single worker has enough contiguous free resources, so a large task waits indefinitely while the cluster reports plenty of headroom. Operators see a cluster at 70% utilisation with tasks queued, which reads as a scheduler bug and is actually a packing consequence. The related trap is over-requesting: tasks that reserve far more than they use make the cluster look full while real utilisation is low, which the Cloud domain covers as requests versus limits.

Retry policy is where a scheduler either contains a failure or amplifies it. A task that fails is re-run; if it fails again it is re-run again, up to a limit. Two things must be true for this to be safe. The retries must be bounded, or a deterministically failing task consumes the cluster forever. And the retries must be attributed, or a single bad machine silently absorbs the whole job.

That second point is the one teams miss. A worker with a failing disk, a corrupt local cache or a misconfigured mount fails every task it is given. The scheduler dutifully re-runs each one elsewhere, where they succeed. The job completes. Nothing is reported as broken — and the job took twice as long, every night, for a month. The fix is per-worker failure attribution and blacklisting: count failures by worker, and stop assigning to one that stands out. Without it, the scheduler’s helpfulness is precisely what hides the fault.

The mirror-image mistake is retrying what cannot succeed. A task that fails because its input is malformed will fail identically on every worker. Retrying it three times costs three times the work and produces the same outcome. Distinguishing *retryable* from *terminal* failures — usually by exception class, not by heuristic — is what keeps a retry policy from becoming a work amplifier, the same reasoning Cap Retries as a Fraction of Traffic, Not as a Count per Request applies to request traffic.