Concept CasesILLUSTRATIVESTAGE-SPECIFICCONTESTED

Case: Implement a Job Queue

Enqueue, dequeue, ack, retry. The state is a job with a status — pending, running, done, failed — and the rule that makes the queue honest: a job can run more than once, so its handler must be idempotent. Derived from "send the confirmation email later", as far as one process justifies.

The move Worked example Next questions

The situation, the reflex, and why it stalls

Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.

The question

Something has to happen later, reliably, outside the request. What is a job queue as a concept, what state does each job carry, and why does "reliably" force a rule on code the queue does not own?

The situation

Checkout should not wait for the confirmation email. You want to "put it on a queue". You have seen queues as a library, a service and a table, and you do not know what any of them promise or what your handler has to promise back.

The reflex

Install a queue library, or point at a hosted one, and call enqueue(sendEmail, order). The email goes out, the checkout is faster, and the queue is a solved problem someone else solved.

Why it stalls

A worker crashes after sending the email and before acknowledging the job. The queue, doing its job, hands the job to another worker; the customer gets two emails. The library's guarantee was "at least once" and nobody read what that obliges the handler to do.

What the reflex produces — and fails to produce

A worker crashes after sending the email and before acknowledging the job. The queue, doing its job, hands the job to another worker; the customer gets two emails. The library's guarantee was "at least once" and nobody read what that obliges the handler to do.
A job fails and is retried immediately, forever, filling the log; or it fails once and is dropped. The retry policy was the library's default, and the concept — what "failed" means, how many times, and what happens then — was never stated.
When the product asks "where is order 4711's email?", there is no state to look at: the job is in the library's store in the library's shape, and the four statuses a job passes through were never named.

Problem→Understand→Requirements→Constraints→Unknowns→Decomposition→Smallest Step→Model→Experiment→Observe→Debug→Learn→Iterate

The move

Precisely enough to apply it to a problem you have never seen — not a slogan.

Define the concept: a job is a unit of work to be done later, exactly as described, by whichever worker takes it, with the queue remembering what has and has not been done. "Remembering" is the state; "whichever worker" and "later" are why a job can be seen twice.
Discover the job's state from its life: pending (enqueued, not taken), running (taken by a worker, not finished), done, failed (attempts exhausted). Add the fields the transitions need — attempts, a lease or taken-at, the last error — and challenge the ones they do not.
Write the operations as transitions — enqueue, dequeue (take), ack (done), fail (retry or exhaust) — and the rules: a job is taken by at most one worker at a time; a running job whose worker vanished is taken again; therefore a job may run more than once, and the handler must produce the same outcome when it does (Finding the State Machine).
Write the examples, including the crash-after-work example, and let it produce the rule the reflex missed. Implement in memory with a single worker, trace the crash, and route the delivery-semantics and idempotency mechanics to Backend and Distributed before adding a second worker (Duplicate Requests).

Meaning and state — the status machine before the fields

A job's life is a small state machine, and the fields fall out of its arrows: the retry arrow needs attempts, the "worker vanished" arrow needs takenAt and a lease, the failed state needs lastError. Drawing the machine first is what stops the state canvas from being a list of plausible columns (Finding the State Machine).

The decomposition splits the concept into the parts the queue owns and the one obligation it hands to the handler. That leaf is the lesson.

Kept: id, payload, status, attempts, takenAt, lastError; queue-level maxAttempts and lease length.
Dropped: result (no V1 operation reads it), priority (no job needs to jump yet), scheduledFor (no job is delayed yet) — each returns with the requirement that reads it.
Derived: "is this job abandoned?" from status = running and takenAt older than the lease — never stored as its own flag.

A job queue, by what must be true

Do work later, reliably, by whichever worker takes it

├Jobs with a status machine— the state the queue remembers
- └pending → running on dequeuetestable After dequeue the job is running with takenAt set and is not returned by a second dequeue.
- └running → done on acktestable After ack the job is done and ack again is rejected.
- └running → pending on fail with attempts lefttestable After fail, attempts is incremented and the job is returned by the next dequeue.
- └running → failed when attempts are exhaustedtestable The third fail on a max of three leaves the job failed with lastError set, and dequeue does not return it.
- └running → pending when the lease expirestestable A job taken and never acked is returned by dequeue after the lease, with attempts incremented.
└At most one worker per running job— the arbitration that makes "whichever worker" safetestable Two dequeues never return the same job while it is running — trivially true with one worker, the linked lesson's subject with two.
└Handlers idempotent on the domain key— the obligation the lease createstestable Running the email handler twice for order 4711 sends one email.

The last leaf is not the queue's code. It is what the queue's honesty about crashes demands of the code that uses it.

Operations, the crash example, and the rule about someone else's code

Five operations, each a transition. The example that matters is the crash after the work and before the ack; it is written as a state change and then encoded as a rule, and the rule is unusual because its code lives in the handler, not the queue. The queue can only make the obligation visible.

Notice the queue's behaviour in the example is correct. Reclaiming the job is what "reliably" means; the duplicate email is the handler failing to meet its side of the contract.

enqueue(payload) → pending, attempts 0. dequeue(now) → the oldest pending, now running with takenAt; nothing when the queue is empty — a normal output.
ack(id) → done; rejected unless the job is running. fail(id, error) → pending with attempts + 1, or failed when attempts would exceed maxAttempts; a backoff before the next dequeue is a V2 field.
reclaim(now) → every running job with takenAt older than the lease goes back to pending with attempts + 1 — the operation that makes the lease real.

Worker dies after sending, before ack

before

job 4711: { status: running, attempts: 0, takenAt: t }; email for 4711: sent

worker dies; reclaim(now) after the lease expires →

after

job 4711: { status: pending, attempts: 1, takenAt: none }; email for 4711: still sent

what changed status: running → pending · attempts: 0 → 1 · takenAt: t → none · the email: unchanged — the queue does not know it was sent, which is why the handler must

Handlers are idempotent on the domain key

rule Running a job twice produces the same outcome as running it once, because the queue may deliver it twice.

↓ becomes validation Before the side effect, check whether it has already happened for this job's domain key (the order, not the job id); record it in the same step as performing it where possible.

↓ becomes code

-- in the handler, not the queue
if emails.alreadySent(orderId): return done
send(email); emails.markSent(orderId)   -- see idempotency lessons for making these one step

Representation, dequeue in pseudocode, the trace, and where it stops

Two structures because two access patterns: pending jobs in FIFO order — a queue, O(1) at both ends — and a map by id for ack, fail and reclaim. An array for both would make dequeue O(n) with a shift and ack O(n) with a scan; annotated, not asserted (Queue and Hash Map in DSA).

The trace runs dequeue in a single worker; its branch is "is there a pending job?". With a second worker the branch is a race, and the lesson stops there on purpose: the mechanism that makes two workers safe — a conditional update, a locked row, a broker's visibility timeout — is the linked lessons' content, arrived at with this example in hand.

dequeue after the crash

inputq.pending = [4711 (attempts 1)]; q.byId[4711].status = pending; now = t + lease + 1
lookupreclaim finds no running jobs; pending is not empty → removeFirst → job 4711
branchpending non-empty → take it (a second worker here would race for the same removeFirst)
mutationjob 4711: status pending → running, takenAt none → now
outputjob 4711 — delivered a second time, as promised; the handler's check decides whether a second email is sent

The queue, by version

V1 — in memory, one worker
The five operations over a FIFO and a map; the status machine as tests, including the crash example. — The transitions and the handler obligation are testable with no infrastructure.
V2 — a table with a status column
One row per job; dequeue is a conditional update; jobs survive a restart; a backoff column. — A restart losing every pending job is the first thing noticed; the table makes "where is 4711's email?" a query (Job Queues in Backend).
V3 — several workers
Dequeue that two workers cannot both win — a locked row skipped by the other, or a lease written atomically; a dead-letter destination for failed jobs. — One worker is measured as too slow; the "at most one worker" rule now needs a mechanism, and exhausted jobs need somewhere to go (A Dead-Letter Queue Is a Workflow, Not a Bin; Work Queues: One Task, One Worker, Competing Consumers in Distributed).
V4 — a broker
Delivery, visibility timeouts, fan-out and cross-service consumption owned by a dedicated system; the status machine unchanged. — Throughput, several services or fan-out are requirements the table cannot meet — and the derivation is what lets you read the broker's guarantees and know what they oblige the handler to do (Where You Put the Acknowledgement Decides Everything).

dequeue and reclaim, single worker

1function dequeue(q, now):
2    reclaim(q, now)
3    if q.pending is empty: return nothing
4    job = removeFirst(q.pending)
5    job.status = running; job.takenAt = now
6    return job
7 
8function reclaim(q, now):
9    for each job in q.byId where job.status == running:
10        if now - job.takenAt > q.lease:
11            job.attempts += 1
12            if job.attempts >= q.maxAttempts: job.status = failed; job.lastError = "lease expired"
13            else: job.status = pending; job.takenAt = none; append(q.pending, job)

reclaim scans running jobs — O(running), fine while workers are few; an index by takenAt is the answer when it is not. Every line encodes an arrow of the status machine.

The implementation ladder

Concept, examples, pseudocode, code, tests, production — for the concept this lesson is about. Code is the fourth tab, not the first.

concept Job Queue advanced

Build it step by step →

Job Queue = A holding place for units of work that will be performed later by a worker, each unit tracked from waiting to finished so that none is done twice at once and none is forgotten.

Identity, ownership, lifetime

Does a job have identity? Yes. Two "send welcome email to alice" jobs are two jobs — or should be one, which is a deduplication rule the queue does not have by default. A job needs an id from the first version because ack and fail have to name one.
Who owns it? The queue owns the job's state; the producer that enqueued it owns its meaning; a worker owns it only while it is running, and that ownership is what "at most one worker at a time" protects.
How long does it exist? From enqueue until it is done or has failed for the last time. Whether a done job is kept for inspection is a retention decision; V0 keeps everything, V2 keeps rows with a status, V4 moves the permanently failed ones to a dead-letter list.
Should it survive reload? Yes — that is most of the point. A job lost when the process restarts is a welcome email nobody gets. The queue is persistent from V2, and "never lost" is only a rule from then on.
Who may run it? Any worker, but only one at a time. Which worker is not part of the job's identity; that a worker holds it is part of its state.

State it must remember

ididkeepack, fail and retry name one job.
typestring — send-email, resize-image, …keepThe worker picks a handler by type.
payloadsmall object whose shape depends on typekeepWhat the handler needs — ids, not whole records, so the job stays small and reads fresh data when it runs.
statuspending | running | done | failedkeepThe state machine. dequeue only takes pending; ack only accepts running.
attemptsinteger ≥ 0keepHow many times it has been started; the retry rule compares it to a maximum.
maxAttemptsinteger > 0dependsWhen to stop retrying.
workerIdid or emptydependsWhich worker holds a running job.
resultanythingdropCallers might want the output.
enqueuedAttimestampderiveFIFO order, and how long jobs wait.
lastErrorstringkeepWhy the last attempt failed, for the operator.

Operations

create Enqueue — the new job
domain Dequeue — the job to run, or nothing when the queue is empty
update Acknowledge — the job
update Fail — the job, with its new status
update Retry a failed job — the job

Rules that must always hold

• A job is running by at most one worker at a time.
• A job is never lost — so it is delivered at least once, and the handler must be idempotent.
• attempts never exceeds maxAttempts.
• Only a running job can be acked or failed.
• Jobs run in the order they were enqueued.

How to do it

Most important first.

Write the meaning and underline "later" and "whichever worker" — the first says state must outlive the request, the second says two workers can disagree about a job unless the queue arbitrates.
Draw the status machine before the fields: pending → running → done; running → pending (retry) or failed (exhausted). Then list the fields each arrow needs: attempts for the retry arrow, takenAt or a lease for the "worker vanished" arrow, lastError for failed (States That Must Be Unrepresentable).
Challenge: payload (keep), status (keep), attempts (keep), maxAttempts (depends — per queue or per job), takenAt (keep — the only way to detect a vanished worker), result (drop in V1 — no operation reads it), priority (drop until a job needs to jump).
Write the operations with their errors: ack on a job that is not running is an error; dequeue on an empty queue is a normal "nothing" output, not an error.
Write the crash example step by step, predict what the queue does, and then write the rule for handlers. Then write the retry example with a maximum, and the dead-letter decision.
Implement dequeue in memory, single worker, and stop before the second worker — its arrival is the trigger for the linked lessons on delivery and storage (Job Queues, At-Least-Once Delivery in Backend).

Worked on a concrete problem

The move has to produce something. This is what it produced.

Meaning: a unit of work to be done later by whichever worker takes it, with the queue remembering what has and has not been done. Identity: every job has one — two "send email for order 4711" jobs are two jobs, which is why idempotency must be keyed by the order and not by the job. Lifetime: from enqueue until done or failed; done jobs may be forgotten after a retention rule.
State per job: id, payload { orderId }, status, attempts, takenAt, lastError. Queue state: the pending jobs in order, a maxAttempts, a lease length after which a running job is considered abandoned. Dropped: result, priority, scheduledFor — each arrives with the requirement that reads it.
Operations: enqueue(payload) → pending job; dequeue(now) → the oldest pending job marked running with takenAt = now, or nothing; ack(id) → done; fail(id, error) → pending with attempts + 1, or failed when attempts reach the maximum; reclaim(now) → every running job whose takenAt is older than the lease goes back to pending.
Rules: a job is running on at most one worker — enforced by the queue, in V1 by the single process; a running job whose lease expired is taken again — so a job may run more than once; a handler must therefore be idempotent for its job's key; attempts never exceed the maximum; ack of a job that is not running is rejected.
Examples: [] → enqueue(order 4711) → [4711 pending, attempts 0]; dequeue → 4711 running, takenAt t; ack → done. The crash: dequeue → running; the handler sends the email; the worker dies before ack; reclaim after the lease → pending, attempts 1; dequeue again → running; the handler runs again — and sends a second email unless it first checks "was the email for 4711 already sent?". The rule came from this example, and it is a rule about code the queue does not own.
Representation: pending jobs in a queue — FIFO, O(1) enqueue and dequeue — plus a map from job id to job for ack, fail and reclaim in O(1) average; reclaim scans running jobs, O(running). In a database: one row per job with a status column and UPDATE … SET status = running, taken_at = now WHERE id = (SELECT id … WHERE status = pending ORDER BY created LIMIT 1 FOR UPDATE SKIP LOCKED) — the second worker's problem, deliberately routed (Work Queues: One Task, One Worker, Competing Consumers in Distributed).

How you know it worked

What now exists that did not before, and what question you can now ask.

The status machine is drawn, every arrow names the field it needs, and there is no state a job can be in that the machine does not show.
The crash-after-work example is written with the queue's correct behaviour and the handler's obligation, and your handler checks before it sends.
You can say what "at least once" means in terms of your own operations — dequeue, lease, reclaim — rather than as a phrase from a README.
You can name what changes when a second worker arrives: the "at most one worker" rule needs a mechanism, and you know which lesson has it.

The questions you can now ask

The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.

Next questions

?Which statuses can a job be in, and which field does each transition need?
?What does the queue do when a worker takes a job and vanishes — and what does that oblige the handler to do?
?What is the key a handler must be idempotent on — the job id or the thing the job is about?
?How many attempts, how long between them, and what happens to a job that exhausts them?
?What changes about "at most one worker per job" when there is a second worker?

What can go wrong

How the move itself fails

The queue is derived and the handler is not: the rule "handlers are idempotent" is written in the lesson and not in the email sender, and the second email goes out exactly as before.
Exactly-once is attempted inside the queue — "just don't reclaim" — and a job whose worker died stays running forever; the lease was the honesty, and removing it made the queue lie instead of duplicating.
Retries are unbounded or immediate; the derivation produced attempts and maxAttempts and the implementation ignored them because the first version had no failures to test against — write the fail example.

What the move costs

A lease and a reclaim make the queue honest about crashes and guarantee that some jobs run twice; the alternative — no reclaim — loses jobs silently, which is worse and looks better.
Idempotent handlers cost a check before every side effect and a record of what was done; for an email that is a row per order, for a payment it is the whole design.
Deriving the queue in memory means one certain rewrite to a durable store when the process restarts — mitigated, as always, by the examples becoming the tests for the durable version.

Misreads

"A queue guarantees each job runs once." A queue guarantees each job is delivered at least once or at most once, and the honest ones say which; "exactly once" is the handler's idempotency plus at-least-once delivery, not a property the queue can promise alone (Where You Put the Acknowledgement Decides Everything in Distributed).
"Idempotency is the queue's job." The queue can deduplicate job ids; it cannot know that two different jobs are about the same order. The handler owns idempotency on the domain key (Job Idempotency in Backend).
"Use a message broker from the start." A table with a status column and a lease is a job queue, and it is the right V1 for a single service with a handful of job types; the broker arrives when throughput, fan-out or cross-service delivery are measured requirements (Message Queues in Architecture).

Where this applies

Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

ILLUSTRATIVEOrder 4711, the confirmation email, the lease and a maximum of three attempts are invented; the crash-after-work interleaving is the real shape behind every duplicate-side-effect incident.
STAGE-SPECIFICFor a single service in V1 an in-memory queue with a single worker, or a table with a status column, is the whole answer; in a system with several services and workers the same state machine holds and the mechanism for "at most one worker" becomes the linked lessons' subject.
CONTESTEDSome practitioners argue that hand-rolling any queue is a mistake because the failure modes — leases, reclaim, retry backoff, dead letters — are exactly what mature brokers have spent years getting right, and a team that derives them will relearn each one as an incident; the strongest form is that the derivation is worth doing on paper and never in production code. This lesson agrees about production and disagrees about paper: the derivation is what lets you read the broker's guarantees and know what they oblige your handler to do.

Where the depth lives

This domain asks the question and hands the answer off by name.

Software Designstate-machines idempotency-by-design

Architecturemessage-queues

Backendjob-queues at-least-once-delivery job-idempotency dead-letter-queues queue-semantics

DSAqueue hash-map

Distributed Systemsdelivery-semantics work-queues idempotent-operations