Worker Pipeline
Case Study: Queue and Worker Pipeline
A data ingestion pipeline: customers submit files of records, each record is validated, enriched against two third-party APIs, and written to a database and to object storage. Volume is spiky and customer-driven — nothing for six hours, then forty thousand records in ten minutes because someone finished an onboarding. This case study is about the operational properties of asynchronous work: how it scales, how it retries, what it does with a job that can never succeed, what identity it runs as, and how you answer "where is my file?".
Requirements
- Accept a submission and acknowledge it immediately, without holding the connection for the duration of the work.
- Process records at a rate that keeps the oldest waiting job under an agreed age, even during a 40,000-record burst.
- Survive worker crashes and third-party API failures without losing work.
- Never let one unprocessable record stop the pipeline.
- Answer "what happened to this specific record?" for any submission in the retention window.
- Run with an identity scoped to what the pipeline actually touches.
Deliberately not requirements
Half of a design is what it refuses to do. These are the refusals.
Out of scope, on purpose
- No ordering guarantee across records — each record is independent, which is what makes this design possible at all.
- No exactly-once delivery: the pipeline is built for at-least-once with idempotent effects.
- No sub-second processing latency; the contract is minutes, not milliseconds.
How the design got here
In order. Each stage leads with the problem that forced it.
Stage 1
Process inside the request
Forced by
The requirement, at the volume it started with. A submission of two hundred records validated and enriched in about four seconds, well within the HTTP timeout. There was no queue, no worker and no state machine, and the whole feature was one endpoint. This is the correct starting design, and skipping it to "build it properly" would have cost weeks before anyone knew whether the feature mattered.
No pipeline at all: the API does the work while the client waits.PROVIDER-NEUTRAL
Customer integrationpublic
Load balancerpublic— 60-second idle timeout — the constraint that ends this stage
API instancesprivate— validates, enriches and writes synchronously
PostgreSQLprivate
Enrichment APIspublic— two third parties, each with its own latency and its own bad days
Customer integration→Load balancer· POST /submissionscrosses boundary
Load balancer→API instances
API instances→Enrichment APIs· enrich each recordcrosses boundary
API instances→PostgreSQL· write results
| Decision | Reason | Alternative | Trade-off |
|---|---|---|---|
| Do the work inline while it fits inside a request. | A queue and a worker pool are two more deployables, a delivery-semantics problem and a status-tracking problem. None of that is justified by four seconds of work. | Build the queue immediately, which is what most teams do and what costs them a fortnight before the first customer has used the feature. | The design has a hard ceiling defined by the load balancer's idle timeout, and the failure at that ceiling is ugly: a timeout after the work has partly happened, with no record of how far it got. |
Stage 2
A queue and one worker
Forced by
A 200,000-row file took eleven minutes. The load balancer closed the connection at sixty seconds, the customer's client treated that as a failure and retried, and two full ingests of the same file ran concurrently against the same third-party rate limit — producing duplicate records and a bill from the enrichment provider.
Acceptance and processing become separate concerns with a durable boundary between them.PROVIDER-NEUTRAL
Customer integrationpublic
API instancesprivate— writes the submission, fans out one message per record, returns 202 with a submission id
Upload bucketprivate— the raw file, uploaded directly with a signed URL
Record queueprivate— at-least-once, visibility timeout sized to the slowest record
Worker (single)private
PostgreSQLprivate— submission and per-record status rows
Enrichment APIspublic
Customer integration→Upload bucket· PUT file (signed URL)crosses boundary
Customer integration→API instances· POST /submissions → 202crosses boundary
API instances→Record queue· enqueue per record
Record queue→Worker (single)· receive
Worker (single)→Enrichment APIs· enrichcrosses boundary
Worker (single)→PostgreSQL· write record + status
| Decision | Reason | Alternative | Trade-off |
|---|---|---|---|
| The API returns 202 with a submission id and does none of the work. | It decouples the client's connection lifetime from the work's duration, which is the entire point. The submission id is what makes the work addressable afterwards. | Keep the connection open with streaming progress, which gives a nicer client experience and reintroduces the timeout ceiling plus a held worker. | The client must now poll or receive a callback, and you have introduced a state machine — accepted, processing, partially failed, complete — that has to be designed, stored and explained in the API contract. |
| One message per record, not one per file. | It makes the unit of retry a single record, so one bad row cannot poison a file of 200,000 good ones, and it lets throughput scale by adding consumers rather than by making one consumer faster. | One message per file, which is far simpler to track and makes retry mean "redo eleven minutes of work" and parallelism mean nothing. | Message volume goes up by four orders of magnitude, which costs money per request and turns the fan-out itself into a job that can fail halfway. The submission needs an expected-record-count so you can detect a partial fan-out. |
| The visibility timeout is sized to the slowest plausible record, not the average. | If the timeout expires while a worker is still processing, the queue redelivers the message and a second worker starts the same record — the classic source of mysterious duplicates. | A short timeout with a heartbeat that extends it, which is more precise and more code to get wrong. | A long timeout means a crashed worker's messages stay invisible for that long before anyone else can pick them up, so crash recovery is slower. You are trading duplicate work against recovery latency, and you must pick a side deliberately. |
Stage 3
Many workers, scaled by queue age
Forced by
One worker cleared roughly 1,200 records an hour. A customer onboarding pushed 40,000 records on a Monday morning, the queue took most of two days to drain, and every other customer's submissions sat behind it — including a 50-record file that should have taken twenty seconds and took nineteen hours.
Throughput becomes a scaling parameter, and fairness becomes a design problem.PROVIDER-NEUTRAL
Standard queueprivate— bulk submissions
Small-submission queueprivate— files under a threshold — the fairness fix
Worker pool (autoscaled)private— scales on oldest-message age; floor of one, ceiling set by the third-party rate limit
Enrichment APIspublic— the real ceiling: a per-account rate limit no amount of workers can exceed
PostgreSQLprivate
Small-submission queue→Worker pool (autoscaled)· higher priority
Standard queue→Worker pool (autoscaled)· bulk
Worker pool (autoscaled)→Enrichment APIscrosses boundary
Worker pool (autoscaled)→PostgreSQL
| Decision | Reason | Alternative | Trade-off |
|---|---|---|---|
| Scale on oldest-message age, not on queue depth. | Age is the customer-visible property. Depth tells you how much work exists; age tells you whether anyone is waiting too long, which is what the contract actually promises. | Depth-based scaling, which is simpler and over-scales for a flood of trivial jobs while under-reacting to a slow backlog. | Age is noisier and meaningless at zero workers, so it needs a floor. It also lags: by the time age is high, the backlog already exists — no reactive policy can scale before demand arrives. |
| A separate queue for small submissions. | A single FIFO queue lets one customer's bulk load block everyone. Two queues with different priorities is the cheapest possible fairness mechanism and solves the actual complaint. | Per-tenant queues or a fair-scheduling consumer, which is genuinely fair and is a scheduler you now own and must debug. | A threshold that must be chosen and will be gamed — a customer who splits a large file into small ones jumps the queue. Fairness by heuristic is fairness until someone notices the heuristic. |
| Cap the worker pool at the third-party rate limit. | Autoscaling always terminates at a fixed dependency. Past the enrichment provider's limit, extra workers produce 429s, retries, and a larger bill for the same throughput. | Scale freely and let retries absorb the rejections, which wastes compute and can get your account throttled harder or suspended. | There is now a hard maximum throughput, so a sufficiently large burst *will* take a long time and no amount of money fixes it. That number belongs in the customer contract, not in a config file nobody reads. |
Stage 4
Retries, backoff and a dead-letter queue
Forced by
Two failures in one week, at opposite extremes. A malformed record threw on every attempt; the queue redelivered it indefinitely, and a worker spent three days crashing and restarting on the same message while appearing perfectly healthy. Meanwhile a thirty-second outage at one enrichment provider caused 4,000 *good* records to be marked permanently failed, because the code treated every exception the same way.
Failure classification made explicit: retry the transient, park the permanent.PROVIDER-NEUTRAL
Record queueprivate— maxReceiveCount = 5, then the message moves on
Workersprivate— classifies: transient → retry with backoff and jitter; permanent → fail the record immediately
Dead-letter queueprivate— messages that exhausted their attempts; alerting on depth *and* on age
Enrichment APIspublic— 429 and 5xx are transient; 400 and 422 are permanent — the distinction the code was missing
PostgreSQLprivate— per-record status with a reason code, so failures are queryable rather than only loggable
Replay jobprivate— after a fix, re-enqueues DLQ messages — because a DLQ without a way out is a bin
Record queue→Workers
Workers→Enrichment APIscrosses boundary
Workers→PostgreSQL· status + reason code
Record queue→Dead-letter queue· after 5 failed receives
Replay job→Record queue· re-enqueue after a fix
| Decision | Reason | Alternative | Trade-off |
|---|---|---|---|
| Classify failures as transient or permanent before deciding to retry. | Retrying a permanent failure burns capacity and never succeeds; failing a transient one throws away work that would have succeeded thirty seconds later. Both incidents in this stage were the same missing distinction. | Retry everything a fixed number of times, which is simple, and permanently costs you a multiple of the wasted work on every malformed input. | The classification lives in your code and drifts from reality as providers change their error semantics. It needs tests and periodic review, and it will be wrong at least once in an interesting way. |
| Exponential backoff with jitter on transient failures. | Without jitter, every worker that failed during the same provider outage retries at the same instant and recreates the overload the moment the provider recovers. | Fixed-interval retries, which are easier to reason about and synchronize your entire fleet into a thundering herd. | Worst-case latency for a record grows with the backoff schedule, so the oldest-message-age signal becomes harder to interpret — some of that age is your own deliberate waiting. |
| A dead-letter queue with alerting on depth and on age, plus a replay path. | Bounded attempts stop the poison-message loop; the alert stops the DLQ becoming a silent data-loss channel; the replay path is what makes fixing a bug meaningful for the records that already failed. | Log the failure and drop the record, which needs no new infrastructure and means "we lost your data" is the design. | Somebody must own the DLQ. An unread dead-letter queue is worse than none at all, because everyone believes it is a control. And replay must be idempotent, or fixing the bug creates duplicates. |
Stage 5
A worker identity of its own
Forced by
The workers used the same static access key as the API, stored in the deploy repository. It could read every bucket in the account. When it appeared verbatim in a support log shared with a customer, rotating it required a coordinated restart of every service at once — a self-inflicted outage on top of a security incident.
Two workloads, two identities, two blast radii.PROVIDER-NEUTRAL
APIprivate
API roleinternal— write to the upload prefix, enqueue to the record queue, read/write submission tables
Workersprivate
Worker roleinternal— receive and delete from the queue, read the upload prefix, write the results prefix, read the enrichment credential — nothing else
Secret managerprivate— holds the enrichment API credentials; every read is audited
Object storageprivate— uploads/ and results/ prefixes with separate grants
Record queueprivate
API→API role· assume
Workers→Worker role· assume
Workers→Secret manager· read enrichment credential
Workers→Object storage· read uploads/, write results/
Workers→Record queue· receive, delete
| Decision | Reason | Alternative | Trade-off |
|---|---|---|---|
| Separate roles for the API and the workers, scoped to prefixes and queues rather than to services. | Least privilege is measured by blast radius (§62). If a worker is compromised, the honest question is "what can it reach?" — and "one queue and two prefixes" is a much better answer than "every bucket in the account". | One shared role for the whole application, which is fewer things to maintain and makes every compromise a full-account compromise. | More policies to write and keep current, and a new failure mode: a legitimately new operation is denied at 02:00 and fails silently, exactly as in break-iam. Least privilege buys blast-radius reduction with a permanent stream of small permission problems. |
| The enrichment credential lives in the secret manager, read at startup by the worker role. | It puts a rotatable, audited boundary around a third-party credential that is also a billing instrument — a stolen enrichment key is somebody spending your money. | An environment variable set at deploy time, which is simpler and makes rotation a full redeploy and leaves the value in the deployment system's history. | The secret manager is now in the worker's startup path, so it needs a private endpoint and a cached fallback, and a secret-service blip becomes a worker that will not start. |
Stage 6
Per-record observability
Forced by
A customer asked where their file was, and the only answer anyone could give was "the queue has 12,000 messages". There was no way to say which of their records had been processed, which had failed, or why — the logs were per-worker and interleaved across every tenant.
Observability designed around the question people actually ask.PROVIDER-NEUTRAL
APIprivate— assigns a submission id and a per-record id; both travel with the message
Workersprivate— every log line carries submission id, record id and attempt number
Status storeprivate— per-record state and reason code — the authoritative answer to "where is my file?"
Metricsprivate— oldest-message age, throughput, per-reason failure counts, DLQ depth and age
Log storeprivate— structured, queryable by submission id
Alertingprivate— pages on queue age past the contract, DLQ growth, and throughput dropping to zero while depth is non-zero
API→Status store· create records as pending
Workers→Status store· transition each record
Workers→Log store· structured lines
Workers→Metrics· counters and timings
Metrics→Alerting
| Decision | Reason | Alternative | Trade-off |
|---|---|---|---|
| Per-record status in the database, not only in logs. | Support needs a query, not a log search. A status row with a reason code answers the customer's question in one lookup and is retained on your schedule rather than the log store's. | Reconstruct status from logs, which is free until the first time you need it under time pressure with a customer on the phone. | One write per record state transition, which at high volume is real database load — this is the stage where the status table can become busier than the data table it describes. |
| Alert on the oldest-message age against the contract, and on throughput-zero-while-depth-nonzero. | Age is the promise you made. The second condition catches the failure this pipeline has that nothing else detects: workers alive, healthy, and processing nothing — a stuck consumer, an exhausted credential, a wrong permission. | Alert on queue depth, which fires during every normal burst and stays silent during a total stall with an empty queue. | Two more rules to tune, and the throughput-zero rule needs a grace period or it pages during every quiet night. Every good alert has a false-positive story. |
What would break this
Every design has a load, a failure or an organization size at which it stops being the right one.
Breaking points
- A requirement for ordering. Independent records are what makes parallelism free; the moment record N must be processed after record N-1, you need partitioned ordering keys, and throughput becomes bounded by the slowest partition.
- A single record that takes longer than the maximum visibility timeout. Past that the queue redelivers work that is still running, and you need checkpointing or a job store rather than a message queue.
- Payload larger than the queue's message size limit. The fix is a pointer to object storage in the message, which adds a lifecycle problem: who deletes the payload after the job succeeds, and after it fails?
- A throughput target above the third-party rate limit. No infrastructure change fixes this; it is a commercial negotiation or a different provider.
- Fairness needs beyond two queues. Real per-tenant fairness means a scheduler, and a scheduler is a piece of software you now own, debug and page on.
- Results that must be queryable as a stream rather than a status row. That is a different data platform, and this pipeline becomes its producer.
Cost shape
Drivers and relative weights. Never a price.
A queue pipeline's bill is mostly compute you can actually turn off — plus a per-message fee nobody models.ILLUSTRATIVE
Worker compute spiky
driven by instance-hours × records per hour, following the burst · The dominant line, and the one autoscaling genuinely controls: idle overnight means near-zero worker spend.
Queue requests · the surpriseusage
driven by send + receive + delete API calls per message, not per job · One record is at minimum three billable operations, and empty long-polling receives count too. At one message per record this line is real, and it is the direct price of the fan-out decision.
Third-party enrichment · the surpriseusage
driven by API calls per record, billed by the provider · Frequently larger than the entire cloud bill, and a retry storm multiplies it. Classify failures properly or pay twice for the same record.
Database fixed
driven by instance-hours plus write volume from status transitions
Object storage usage
driven by gigabytes of uploads and results retained · A lifecycle rule that expires raw uploads after the retention window is the cheapest cost control in this design.
NAT processing · the surpriseusage
driven by per-GB on every outbound enrichment call
Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.