FilesGENERALRUNTIME-SPECIFICCLOUD-SPECIFIC

What Happens After the Bytes Land

Stored is not ready. Scanning, transcoding and thumbnailing are background work with a state machine, and skipping the state machine is how unscanned files get served.

What actually happens How to build it

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.

The question

A file is in the bucket. What still has to happen before anyone should be allowed to use it?

The requirement

Uploaded attachments must be virus-scanned before any other user can download them, images need a thumbnail, and PDFs need a page count and extracted text for search.

The obvious build

Do it in the upload handler. The file is right there — scan it, resize it, extract the text, save the results, return the finished attachment. One request, one consistent result.

Why it breaks

A 20 MB PDF takes seconds to parse and a video takes minutes to transcode. The client is waiting on a request that will time out at a proxy long before the work finishes (Timeouts).

How it breaks in production

A 20 MB PDF takes seconds to parse and a video takes minutes to transcode. The client is waiting on a request that will time out at a proxy long before the work finishes (Timeouts).
Image and video processing is CPU-bound. On a single-threaded event loop it blocks every other request in the process; on a thread-per-request model it consumes a worker for the duration (Blocking the Event Loop).
The scanner is a separate service that can be slow or down. Its availability is now your upload endpoint's availability (Calling Something You Do Not Control).
If processing fails after the file is stored, you have an object, a row, and no defined state — so the frontend shows an attachment whose thumbnail never appears and whose scan never ran.
With direct uploads the bytes never passed through your process at all, so there is no handler to do this work in (Presigned URLs).
Memory: decoding a 12,000×12,000 pixel image allocates hundreds of megabytes regardless of how small the compressed file was.

Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale

What is actually happening

Post-upload work is background work. The upload records that an object exists; a job does everything else, and the file carries an explicit status that the rest of the system reads (Background Jobs).
The status is a small state machine, and its value is that it makes "not usable yet" and "will never be usable" different from each other: pending → processing → ready, with failed and rejected as terminal states. rejected (a virus, an unsupported format) is a normal outcome, not an error.
Access control keys off the status. Nothing outside the uploader may read an object that is not ready — that single rule is what makes scanning meaningful, and it must be enforced in the serving path, not in the UI (Serving Files).
The job is triggered either by the upload handler enqueuing it, or by a storage event for direct uploads. Storage events are at-least-once and can arrive out of order or for objects you did not expect, so the handler must be idempotent and must validate that the object matches an intent it authorized (At-Least-Once Delivery).
Processing is untrusted-input handling in its purest form: you are handing attacker-supplied bytes to an image decoder, a PDF parser or an archive extractor, all of which are large C libraries with a long history of memory-safety bugs. Isolation and resource limits are part of the design, not hardening added later (Sandboxing Untrusted Workloads).
Every processing step has bounds that must be explicit: maximum decoded pixel count, maximum archive expansion ratio and entry count, maximum wall-clock time, maximum memory. Unbounded work on attacker-supplied input is the whole vulnerability class.
Results are new objects and new rows: a thumbnail is its own object with its own key, and the extracted text is a database column or a search-index document (Keeping a Search Index in Sync).

Stored, then a state machine

The single most valuable artefact in this lesson is the status field. It turns "is this file usable" from an inference — the object exists, so presumably yes — into a fact the serving path can check.

Note that rejected is a normal terminal state, not a failure. A virus-infected upload processed correctly ends in rejected; the system worked. Conflating it with failed means either retrying a scan on a known-bad file forever, or treating an infrastructure failure as a content decision.

pending — exists, readable only by the uploader. The default, and the safe one.
processing — a worker holds it; a deadline makes a crashed worker visible.
ready — scanned, derivatives durable, and only now readable by others.
rejected — a correct outcome. Tell the user why; do not retry.
failed — infrastructure, not content. Retry with backoff, then dead-letter (Dead-Letter Queues).

UserLLMAgentToolDataDecisionHumanGuardrail

Bounded, isolated, and suspicious of every byte

SIMPLIFIEDSandboxing is shown as an assumption rather than implemented. In practice it is a separate container or microVM with dropped capabilities, no network egress, a read-only filesystem and hard rlimits — and the details are platform-specific enough to be their own topic (Sandboxing Untrusted Workloads).

Processing hands attacker-supplied bytes to decoders that are among the most exploited libraries in software. The two controls that matter are running that code somewhere it cannot hurt you, and refusing inputs whose *decoded* cost is unreasonable — which is not the same as their uploaded size.

Bounds before work, not after

1MAX_PIXELS = 50_000_000        # decoded, not file size
2MAX_ZIP_RATIO = 100            # expanded / compressed
3MAX_ENTRIES = 1_000
4DEADLINE_SECONDS = 60
5 
6def process(key: str, expected: Intent) -> Status:
7    # 1. the event names a key; only an intent we issued makes it legitimate
8    if expected is None or expected.key != key:
9        return Status.IGNORED          # not something we authorized
10 
11    head = storage.head(key)           # size and type from storage, not the client
12    if head.size > expected.max_bytes:
13        return Status.REJECTED
14 
15    # 2. scan first. an error here is NOT clean.
16    verdict = scanner.scan(key, timeout=DEADLINE_SECONDS)
17    if verdict is Verdict.INFECTED:
18        return Status.REJECTED
19    if verdict is Verdict.ERROR:
20        raise TransientError("scanner unavailable")   # retry; stays unreadable
21 
22    # 3. bound the DECODED cost before decoding
23    w, h = probe_dimensions(key)       # header only; no full decode
24    if w * h > MAX_PIXELS:
25        return Status.REJECTED
26 
27    thumb_key = f"{key}.thumb.webp"    # derived, still tenant-prefixed
28    render_thumbnail(key, thumb_key)   # in a sandboxed subprocess, hard rlimits
29 
30    # 4. derivatives durable BEFORE the status flips
31    db.attachments.mark_ready(expected.id, thumb_key=thumb_key)
32    return Status.READY

Four properties, each guarding a distinct failure: the event is checked against an intent so an attacker who can write to the bucket cannot inject work; the scanner's *error* raises rather than continuing, so failure leaves the file unreadable; the pixel bound is checked from the header before any full decode allocates memory; and the status flips only after the derivative is durable, so no reader can observe ready with a missing thumbnail.

Where the pipeline breaks

Almost every failure here ends in one of two places: a file stuck in a non-terminal state forever, or a file that became readable without being checked. The first is a support problem; the second is the breach.

Processing failures and responses

Trigger	Symptom	Cause	Response
Enqueue happens before the transaction commits	Worker picks up a job for a row that does not exist yet, or ever	Two writes with no atomicity	Transactional outbox, or trigger from the storage event instead of the handler (The Transactional Outbox).
Storage event dropped or never configured	Files silently stay `pending`	The only trigger was best-effort	A sweeper that finds objects with no processing record and enqueues them.
Scanner times out	Elevated latency, then a decision	An external dependency in the critical path of a security check	Retry with backoff and leave the file unreadable. Never treat an error as clean (Retries).
A file crashes the decoder every time	Worker restarts in a loop; the queue stalls behind it	No attempt limit, no isolation	Bounded attempts, then dead-letter; run decoding in a subprocess so a crash kills the child (Dead-Letter Queues).
Zip with a 1000:1 expansion ratio	Disk full on the worker	Extraction with no output bound	Cap expanded bytes, entry count and nesting depth; reject on exceeding any of them.
Duplicate storage event	Two thumbnails, two search documents	Non-idempotent job	Key the job by object key; use a conditional update so only one attempt flips the status (Job Idempotency).
Status flipped before derivatives are written	Broken thumbnail for the first readers	Ordering	Write derivatives, verify, then flip status — the flip is the commit point.
Backlog grows faster than workers drain	"Ready" takes hours; users report the feature is broken	Fixed worker pool against a bursty producer	Autoscale on queue age, cap per-tenant concurrency, and surface the wait in the UI (Queue Backlog).

How to build it

Most important first.

Give every uploaded file a status column from day one, and make the serving path check it. Adding it after files are already downloadable means a backfill and a window (Schema Migrations from the Application Side).
Enqueue processing rather than doing it inline, and return the pending resource. The API contract should say the attachment is not ready yet rather than pretending it is (The Async Job Pattern).
Make the job idempotent, keyed by object key or content hash, so a duplicate storage event reprocesses harmlessly (Job Idempotency).
Run decoding and extraction in an isolated worker — a separate process, container or sandbox — with hard CPU, memory and time limits, and treat a crash as a rejected file rather than an incident (Worker Processes).
Bound everything on attacker-supplied input: decoded dimensions, archive expansion, nesting depth, page count, duration. Reject rather than truncate, so the limit is visible to the user.
Use a dedicated queue and worker pool for processing so a backlog of large files does not delay unrelated jobs, and cap per-tenant concurrency (Bulkheads).
Route permanent failures to a dead-letter queue with the object key, and make failed a state a human can act on rather than a row that silently never becomes ready (Dead-Letter Queues).
Reconcile: objects with no processing record, and records stuck in processing past a deadline. Both are invisible to request-level monitoring.

What can go wrong

Failure modes

Files stuck in pending forever because the trigger was lost — the enqueue happened after a failed commit, or the storage event was dropped (The Dual Write Problem).
A poison file that crashes the worker on every attempt, taking the rest of the queue with it if there is no attempt limit (Dead-Letter Queues).
Decompression bombs and pixel bombs: small inputs that expand to fill memory or disk during processing.
A scanner timeout treated as "clean", so unscanned files become downloadable — failing open on the one check whose entire purpose is to fail closed (Fail Open vs Fail Closed).
Duplicate storage events producing duplicate thumbnails and duplicate search documents (Webhook Idempotency).
A processing backlog that grows faster than workers drain it, so "ready" takes hours and users assume the feature is broken (Queue Backlog).
A thumbnail written to a key derived from the original in a way that collides across tenants.
Extracted text containing untrusted content that is later rendered without escaping — the file becomes an injection vector through the search index (Cross-Site Scripting (XSS)).

What can race

A user requesting a file between object creation and processing completion. The status check is what makes this deterministic instead of a timing question.
Two storage events for the same object processed concurrently, producing duplicate thumbnails or two search documents unless the job is idempotent and takes a lock or a conditional update (Job Idempotency).
A file deleted while a processing job holds its key: the job must handle a missing object as a normal terminal outcome, not an error to retry forever.
Status flipped to ready before the derived objects are durably written, so the first reader gets a broken thumbnail — write the derivatives first, then the status (Where the Transaction Boundary Goes).

Security

If the serving path does not check status, an attacker gets to upload malware and have your product distribute it to other users during the window before scanning completes — and the window is unbounded if the job never runs.
If a scanner error is treated as clean, an attacker gets a reliable bypass: make the scanner slow or unavailable, then upload. Scan failure must leave the file unusable.
If processing runs in the main application process, an attacker gets a memory-safety exploit surface with your service's privileges and credentials — image and PDF libraries are a recurring source of remote code execution, and they are being fed attacker-chosen bytes (Sandboxing Untrusted Workloads).
If decoded size is unbounded, an attacker gets denial of service from a small upload: a highly compressed image or a nested archive expands to exhaust memory or disk.
If the job trusts the storage event's key without checking it against an intent your API authorized, an attacker who can write anywhere in the bucket gets your pipeline to process — and publish — objects you never accepted (The Trust Boundary).
If extracted text or metadata is stored and later rendered without escaping, an attacker gets stored cross-site scripting through a path nobody thinks of as user input.
If processing workers hold broad storage credentials, an attacker who achieves code execution in a decoder gets whatever those credentials allow — scope them to the prefixes the job touches (Least Privilege in Infrastructure).

Misreads

"The file is uploaded, so it is available." Uploaded and usable are different states, and conflating them is how unscanned files get served (Serving Files).
"Scanning is quick, we can do it inline." Scanner latency is not yours to control, and its availability becomes your endpoint's availability.
"The queue guarantees the job runs." It guarantees delivery of a message that was successfully enqueued. If the enqueue and the database write were not atomic, the message may not exist (The Transactional Outbox).
"Only the file bytes are untrusted." Filenames, metadata, EXIF fields and extracted text are all attacker-controlled and all end up somewhere that renders them.
"We resize images, so we do not need to validate them." Handing bytes to a decoder *is* the risky operation — resizing is the attack surface, not the mitigation.

Operating it

How you see it in production

Track the status distribution over time. A rising pending count is a broken trigger; a rising failed count is a broken processor or a new file type (The Metrics a Backend Must Emit).
Measure time from object creation to ready, as a distribution. It is the user-visible number and it lives entirely outside your request metrics (Depth Is Not an Emergency; Age Is).
Alert on files stuck in processing past a deadline — a crashed worker leaves rows in a state nothing else will move.
Count scanner outcomes separately: clean, infected, error. The error rate is the one that becomes a security problem if anyone decides to treat it as clean.
Log worker resource usage per file and alert on OOM kills in the processing pool — that is a decompression bomb or an oversized image, not a capacity problem (OOM Kills and CPU Throttling).
Reconcile objects in the bucket against processing records so silently unprocessed files are found by a job rather than by a customer.

What changes at 10x and 100x

Processing is the part that scales with CPU rather than with request rate. Workers scale independently of the API, which is the main structural reason to separate them (Worker Scaling).
At 10x, the queue absorbs bursts and latency to ready grows. Whether that is acceptable is a product question, and it should be answered before the burst.
At 100x, transcoding cost dominates everything else in a media product, and the interesting optimizations are doing less work — fewer renditions, on-demand instead of eager, caching aggressively (Cache-Aside).
Per-tenant fairness matters here more than in most queues: one customer uploading ten thousand files should not push everyone else's thumbnails behind them (Multi-Tenancy).

What this costs

Asynchronous processing means the user does not get a finished result. The UI has to represent a pending state, and clients must poll or subscribe — real product cost for real reliability (How the Client Learns the Job Finished).
A status state machine touches every read path. Every query for a file now has a status condition, and forgetting it in one place is the vulnerability.
Sandboxed workers cost operational complexity: another deployable, another scaling dimension, another thing to monitor. They also mean a decoder exploit does not reach your database credentials.
Strict bounds reject files users legitimately have, and each rejection is a support conversation. Generous bounds are an availability risk.
Retries fix transient failures and re-run expensive work; without idempotency they also duplicate outputs.

Where this applies

Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.

GENERALThe status state machine, idempotent jobs and bounded processing apply to any stack and any storage provider.
RUNTIME-SPECIFICWhere inline processing hurts differs: on Node, a synchronous image decode blocks the single loop thread and stalls every in-flight request in that process; on a pre-fork Python or Ruby app server it occupies one worker of a small fixed pool; on the JVM or Go it consumes one of many threads and degrades more gradually. The conclusion — do not do it in the request — is the same, but the symptom you will see first is not (Blocking the Event Loop).
CLOUD-SPECIFICStorage-event delivery differs in ways that change the code: S3 event notifications to SQS or Lambda, GCS notifications to Pub/Sub, Azure Blob events to Event Grid — different payload shapes, different ordering guarantees, and different behaviour on repeated delivery. Managed scanning and transcoding services also differ in whether they can gate access until a scan completes, or only report a result afterwards.

Where the depth lives

This domain teaches the application-side mechanism and hands the rest off.

API Designasync-job-pattern job-completion-notification

Securityfile-upload-security sandboxing fail-open-vs-fail-closed

Performancequeue-age

Cloudobject-storage oom-and-cpu-throttling

Concurrencybounding-concurrency

Domains that do not exist yet

— Media engineering — codec choice, rendition ladders and adaptive bitrate packaging are their own discipline; this lesson only covers the pipeline that invokes them.