Scheduler vs Orchestrator
"Run at 02:00" versus "run B after A succeeds, retry C, skip D if the source is empty" — a difference in what the system remembers, not in how it is configured.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
When does a schedule stop being sufficient, and what exactly does an orchestrator add beyond firing a command at a time?
The on-call engineer at 07:00 who has to answer "is today's data complete, and if not, which of these tables is affected". A scheduler can only tell them what started. An orchestrator can tell them what succeeded, what is waiting, and what was skipped.
A scheduler's unit is the invocation — one command, one moment, no identity beyond its timestamp. An orchestrator's unit is the task run — a task, an interval, a status, an attempt number and a history. Almost every difference between them follows from that change of unit.
Cron plus a shell script. It is the correct starting point far more often than architecture diagrams admit: no infrastructure, no database, no upgrade path, and behaviour that any engineer on the team can read in ten seconds. Most pipelines begin here and a good number should stay.
The first conditional appears. "Only load if yesterday's extract completed" cannot be expressed in a crontab, so it gets expressed in the script — usually as a check for a marker file, sometimes as a grep of a log. That check is now the most important piece of correctness logic in the pipeline and it is untested.
- The first conditional appears. "Only load if yesterday's extract completed" cannot be expressed in a crontab, so it gets expressed in the script — usually as a check for a marker file, sometimes as a grep of a log. That check is now the most important piece of correctness logic in the pipeline and it is untested.
- The marker file survives a partial run. The extract wrote half its rows, crashed, and the wrapper had already created
_SUCCESS— or worse, the object store made the marker visible before the data files (Atomic Publish). - Retries appear next: a
forloop with asleep. It has no jitter, no cap on total attempts and no record of having retried, so a source outage becomes a synchronised hammering from every job on the box (Retry Storms: The Load You Generated Yourself, Without Jitter, Every Client That Failed Together Retries Together). - Then comes the question nobody can answer: "did the 3rd of March load correctly?" There is no history. The evidence is whatever the log rotation kept, and log rotation kept a week.
- Finally, two jobs must both wait for a third. Cron has no way to express a join in the graph, so the third job's script starts calling the other two directly, and the dependency structure disappears into the leaves (Task Dependencies).
What is actually happening
- A scheduler is a function of time: given a clock, emit an invocation. It is stateless by construction — cron does not know whether the last run succeeded, and cannot, because that is not information a clock has.
- An orchestrator is a function of time and state: given a clock and a durable record of every prior (task, interval) outcome, decide which tasks are now eligible. Every capability people associate with orchestration — conditional execution, retries with history, backfill, catch-up, SLA tracking — is downstream of having that record.
- This is why the boundary is so easy to cross without noticing. The moment a script asks "did the other thing work", it needs state; if the system does not provide it, the script invents it, usually as a file whose semantics nobody wrote down (Job Idempotency).
- A hand-built version is not merely less featureful; it is differently wrong. It typically has no notion of an interval, so it cannot answer "run this for last Tuesday". It has no concurrency control, so an overrun overlaps. And its state is a side effect in a filesystem rather than a transactional record (Transactions and ACID).
- Orchestrators also separate the decision from the execution, which a cron entry cannot: the process that decides is not the process that works, so a long task does not delay unrelated scheduling decisions (A Task Is Not a Thread).
The line, drawn precisely
The sharpest way to state the difference is by what the system can be asked. A scheduler can be asked "what time is it". An orchestrator can be asked "what happened". Everything else follows: retries need to know how many attempts have occurred, backfills need to know which intervals are missing, and conditional branches need to know an outcome. None of those is answerable from a clock.
The practical consequence is a trigger rule you can apply without argument. The moment a pipeline needs to express *only if*, it needs state, and the only question left is whether that state will be a designed component or an accident in the filesystem. The accident version is what the "worse" column below describes, and it is extremely common because each individual step in building it is reasonable.
Read the because line as the reason this is not a matter of taste. A wrapper script can be made to work; what it cannot be made to have is a transactional record of outcomes, which is the thing every later requirement turns out to need.
The extract writes `/data/raw/2026-03-11/_SUCCESS` when it finishes. The transform's wrapper script runs at 03:00, tests for that file, and exits quietly if it is absent. A retry loop with a fixed sleep is added later. A lock file is added after the first overlap.
The transform task declares the extract task for the same interval as an upstream dependency. The orchestrator holds the outcome of every (task, interval) pair transactionally, will not start the transform until the extract for that exact interval is in a success state, applies the retry policy attached to the task, and refuses to start a second run of the same interval while one is active.
The marker file conflates three separate facts — the process finished, the data is durably visible, and the data passed validation — into one filesystem entry with no atomicity guarantee and no interval identity. When it is stale, absent, or written by a partially failed run, the check silently returns the wrong answer and the pipeline proceeds. Recorded state distinguishes those facts, is scoped to an interval, and can be queried afterwards, which is what makes the failure diagnosable rather than merely survivable.
Capability by capability
It is worth going through the capabilities individually, because teams usually migrate for one of them and then discover they needed four. The pattern is consistent: each row is trivially available once durable run state exists, and each row requires a bespoke, untested mechanism without it.
The last column is the one that matters during an incident. A capability you do not have is not neutral — it becomes a manual procedure performed under pressure by whoever is awake, and manual procedures performed under pressure are how a partial failure becomes a data corruption.
| Capability | Cron + shell | Orchestrator | What its absence costs you |
|---|---|---|---|
| Run at a time | Native, and genuinely fine | Native, usually the same cron syntax | Nothing — this is the part cron does well |
| Run after another job succeeded | A marker file or a log grep, hand-written per pipeline | A declared edge, evaluated against recorded state | Silent processing of incomplete input, which succeeds and publishes |
| Retry with backoff | A loop with a sleep, no attempt history | A policy on the task, with attempts recorded | Synchronised retry storms and no evidence that a task always fails once |
| Skip on a condition | An early exit 0, indistinguishable from success | An explicit skipped state that downstream rules can read | A skipped day looks exactly like a successful empty day |
| Run one past interval on demand | Only if the script accepts a date argument, which it usually does not | Clear the state for that interval and let it re-derive | Backfills become code edits, and code edits under pressure become incidents |
| Know what ran last month | Whatever log rotation kept | A queryable run history per task and interval | No way to answer "was the 3rd of March correct" after the fact |
| Prevent overlap | A lock file, if someone remembered | A concurrency limit on the task | Two processes writing one output path, producing a state neither would alone |
| Report end-to-end lateness | Not available | A deadline per task or per graph, measured against recorded starts | Freshness is discovered by a consumer instead of by a monitor (The Freshness SLO) |
How the hand-built version fails
None of these failures require an exotic setup. Each one is a normal consequence of building coordination out of files and exit codes, and each has been the root cause of a real data incident many times over. They are grouped here because the response column is the interesting part: in most cases the immediate fix is easy and the underlying problem is that the mechanism cannot express what was meant.
Look at the causes as a set. Every one of them is an attempt to store state — completion, attempt count, exclusivity — in a medium that has no transactions and no notion of which interval it refers to. That is the structural claim of this lesson, and it is why "we will add a check for that" tends not to converge.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Extract crashes after writing some files but after its wrapper touched the marker | Transform runs, succeeds, and publishes a day that is missing several hours | The marker records process completion, not data completeness, and was written outside any transaction | Write the marker only after validation and after the data is durably visible; better, replace it with a recorded run state and a row-count assertion |
| Yesterday's marker was never cleaned up | Transform proceeds on a day whose extract never ran, silently reprocessing yesterday's files | The marker has no interval identity, so a stale success is indistinguishable from a current one | Scope every completion signal to an explicit interval, and make the consumer assert the interval it expects rather than the existence of a file |
| Source is briefly unavailable at 02:00 | Every job on every host retries in lockstep and the source stays unavailable for longer | A fixed-sleep retry loop with no jitter and no global cap on concurrent attempts | Exponential backoff with jitter and a bounded attempt count, expressed as a policy rather than per script (Without Jitter, Every Client That Failed Together Retries Together) |
| A run takes longer than the gap to the next schedule | Two processes write the same output path; the result matches neither run | Cron has no concept of a previous invocation still being active | A lock with an owner and a timeout as a stopgap; a task-level concurrency limit as the real fix |
| A job exits 0 on an empty result to avoid noisy alerts | Days with no data and days with a broken query look identical downstream | Exit code is a single bit being asked to carry three outcomes: success, skip and empty | Distinguish skipped from succeeded explicitly and alert on unexpected emptiness (Volume Anomalies) |
| The box running cron is replaced during a migration | A set of jobs stops running entirely and nothing fires | Schedules were host state, not declared configuration, and absence produces no signal | Monitor dataset freshness independently of the pipeline, so "nothing ran" is detectable at all (Freshness Checks) |
How to build it
Most important first.
- Use a scheduler while the answer to "what does this depend on" is "nothing". That is a real and common situation, and importing a platform to manage it is a cost with no matching benefit.
- Move the moment a conditional appears. The rule is sharp: when a job must ask about another job's outcome, you have started writing an orchestrator. Write it deliberately or adopt one, but do not leave it as a grep in a wrapper script.
- If you must stay on cron for a while, make the boundary explicit: one marker written atomically after validation, checked with a documented timeout, and a lock file that prevents overlap. Those three are the minimum viable orchestrator and they should be written once and shared, not per script.
- Whatever you use, bind runs to intervals from the first day. A pipeline that never knew what
2026-03-11meant cannot be given the concept retroactively without rewriting every task (Incremental Processing). - Keep run history somewhere durable even under cron — a table of (job, interval, started, ended, status, rows) written by the job itself costs almost nothing and answers the question that log rotation destroyed (Pipeline Metrics).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Cron guarantees an invocation attempt at a time, on a host that is up. It does not guarantee the previous invocation finished, that the command exists, or that anyone will learn it failed.
- An orchestrator guarantees that declared dependencies are respected and that outcomes are durably recorded. It does not guarantee the task did the right thing, and it does not guarantee scheduling on time under worker contention.
- Neither guarantees once-only effects. Both can execute a task twice — cron by overlapping, an orchestrator by retrying — and in both cases the property that saves you belongs to the task (Idempotency Keys: The Mechanism).
- A marker file guarantees that something wrote a marker. Whether the data behind it is complete is a separate claim that the marker cannot make unless it is written after validation and after the data is durably visible.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The distinguishing check is a dependency assertion inside the consumer: before reading, verify that the upstream partition for this interval exists, is marked complete, and has a row count within its historical band. This works under either mechanism and is what a cron-based pipeline is missing.
- It misses an upstream that is complete and wrong, and it misses the case where the marker was written by a previous day's run and never cleared — a stale success indicator reads exactly like a fresh one (Reconciliation).
- Under an orchestrator it also misses the undeclared dependency: the assertion only covers inputs you thought to name, which is the same blind spot the graph has.
- Time-chained cron pays a padding tax: the gap between jobs must exceed the worst case of the previous one, so every run is as slow as the slowest night even when it is fast.
- Dependency triggering converts that padding into responsiveness, but it also removes a buffer that was accidentally absorbing variance. Pipelines often get *less predictable* immediately after the move, because they now finish at a variable time instead of a fixed one (Pipeline SLOs).
- Neither approach improves the freshness of the underlying source. If the extract can only run once the source system finishes its own nightly close, the pipeline's floor is that close, and no scheduler changes it.
- Moving from cron to an orchestrator is a semantic migration, not a lift and shift. Scripts that compute their range from
now()must be changed to accept an interval, and until they are, the orchestrator is a scheduler with a nicer UI (Idempotent Data Pipelines). - Historical run records do not migrate. Whatever you knew about past runs stays in the old system, so plan for a period where "has this interval been processed" has two possible sources of truth.
- Adding a dependency to an existing graph changes when downstream tasks run, which changes the freshness that consumers have quietly built expectations around. That expectation is a contract even when nobody wrote it down (Data Contracts).
- Under cron, recovery is manual: work out the range, invoke the script with the right arguments, and hope the script accepts arguments. If it derives its range from the clock, recovery for a past date is impossible without editing code.
- Under an orchestrator, recovery is clearing the state for a bounded set of (task, interval) pairs and letting the graph re-derive. This is only safe if tasks overwrite their interval rather than append to it (Full Refresh vs Incremental).
- In both cases, verify the completeness marker mechanism before re-running: a re-run that fails halfway must not leave a marker claiming success, or the next consumer will read a partial period as a real one.
- Keep the ability to run a single task for a single interval from the command line. It is the tool you will want at 03:00, and platforms that only support "trigger the whole DAG" turn a one-partition fix into a full recompute (Backfills).
What can go wrong
- A cron job overlaps itself and two writers race on one output path.
- A marker file written before the data is durably visible, so downstream reads an incomplete directory (Object Storage as Data Infrastructure).
- A retry loop with no jitter turning a brief source outage into a sustained load spike (Without Jitter, Every Client That Failed Together Retries Together).
- A hand-rolled dependency check that greps a log file whose format changed, so it silently always passes.
- The orchestrator adopted, the scripts unchanged, and every task still computing its own window from
now()— the migration that looks complete and changed nothing. - The mitigation failing: an SLA miss alert configured on the DAG, while the actual failure was that the DAG was paused during an incident and never resumed (Alerts Worth Waking Someone For).
- "We use Airflow, so we are not using cron." The schedule expression is usually still cron. The difference is the state, not the syntax.
- "Cron is for amateurs." Cron is the correct answer for an independent job, and pipelines that adopt a platform for three unrelated scripts pay for capability they never use.
- "A
_SUCCESSfile means the data is complete." It means a process wrote a file. Whether it wrote it after validation, and whether the object store made the data visible before the marker, are separate questions that decide whether the marker means anything (Atomic Publish). - "An orchestrator prevents duplicate runs." It prevents overlapping runs when you configure it to. Duplicate *effects* come from retries, and only idempotent tasks are immune (Job Idempotency).
Operating it
- Whether anything ran at all. Absence of runs is the failure both mechanisms are worst at reporting, and the only reliable detector is a freshness check on the output dataset (Freshness Monitoring).
- Run history per interval — status, duration, rows written. Under cron this must be written by the job; under an orchestrator it is free and underused.
- Overlap: concurrent executions of the same logical job. Easy to detect with a lock table and nearly invisible without one (Reasoning About Races: A Method, Not an Instinct).
- Gap between scheduled time and actual start, which separates "the platform is busy" from "the task is slow" (Depth Is Not an Emergency; Age Is).
- At 10x jobs, cron's failure is organisational rather than technical: nineteen entries across four hosts, and the dependency graph exists only in the start times.
- At 100x, the orchestrator's metadata database becomes a real operational concern, and run-history retention becomes a decision rather than a default.
- Team count scales the problem faster than job count does. Two teams sharing one crontab is a merge conflict; twenty teams sharing one DAG file is a permanent one (Data Platform Engineering).
- A crontab costs nothing to run and an increasing amount to reason about. The cost is engineering time spent reconstructing dependencies during incidents, and it is invisible on any infrastructure bill.
- An orchestrator costs a persistent scheduler process, a metadata database and an upgrade cadence. Those are real and modest; the cost that surprises teams is the metadata store growing with run history and needing its own retention policy.
- The largest cost either way is repeated work caused by coarse triggering — a full rebuild fired by a schedule when only one interval changed (Compute Waste).
- Cron buys simplicity and costs you every question that begins "did". Orchestrators buy those answers and cost a platform, a learning curve, and a new single point of failure that is now upstream of everything.
- Explicit dependencies make the pipeline correct and slower to change: adding a task in the middle of the critical path now visibly costs every consumer, which is honest and unpopular.
- Keeping run history is what makes incidents diagnosable and is also a table that grows forever and eventually needs pruning — the smallest possible instance of a retention decision (Data Retention).
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe distinction is about durable state over prior outcomes, not about any product. A managed scheduler that also records run outcomes and supports dependencies is an orchestrator regardless of what its documentation calls it.
- SCALE-SPECIFICOne job with no dependants belongs on cron and gains nothing from a platform. The threshold is crossed by the first conditional dependency, not by data volume — a terabyte-scale job with no dependants still does not need an orchestrator.
- ORG-SPECIFICWith one team, an undocumented dependency graph lives in someone's head and works. With several teams, the same graph is the coordination problem, and the orchestrator is valuable mostly because it makes dependencies visible to people who did not write them.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the deeper version of "the schedule was host state": configuration that exists only on a machine disappears when the machine does, and the answer is declarative infrastructure rather than a better crontab.
- — Distributed Systems owns why a lock file is not a distributed lock — it has no lease, no fencing token and no way to detect that its holder died — and why the stopgap in the failure table above is a stopgap.