FilesGENERALFRAMEWORK-SPECIFICRUNTIME-SPECIFIC

File Uploads Through the Backend

The obvious path — client to backend to storage — makes your request handler a bandwidth-bound file mover, and it is still the right answer sometimes.

What actually happens How to build it

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.

The question

What actually happens when a user uploads a file to your API, and what does routing those bytes through your process cost?

The requirement

Users attach documents to a support ticket. PDFs and images, usually under a megabyte, occasionally a 200 MB video from a phone. The file must be stored durably and linked to the ticket.

The obvious build

Accept a multipart/form-data POST, let the framework parse it into a file object, validate it, then write it to storage and save the URL on the ticket. One endpoint, one transaction, everything in one place.

Why it breaks

The framework buffers the whole body before your handler runs. Ten concurrent 200 MB uploads is 2 GB of memory in one process, and the process is killed by the OOM killer — taking every unrelated in-flight request with it (Memory Leaks in Backend Services is the usual misdiagnosis; this is not a leak).

How it breaks in production

The framework buffers the whole body before your handler runs. Ten concurrent 200 MB uploads is 2 GB of memory in one process, and the process is killed by the OOM killer — taking every unrelated in-flight request with it (Memory Leaks in Backend Services is the usual misdiagnosis; this is not a leak).
A slow mobile connection holds a worker or a request slot for the entire upload. On a runtime with a bounded worker pool, a handful of slow uploaders is a full outage for everything else (Backend Runtime Models).
The upload occupies bandwidth twice: client to you, then you to storage. You pay for both and the user waits for both, sequentially, unless you stream.
A proxy in front of you has its own body-size limit and its own timeout. A 200 MB upload is rejected with a 413 or 504 that your application never sees and cannot explain (The Request Lifecycle).
Retrying a failed upload means re-sending the whole file. At 200 MB on a phone, that is a feature users experience as broken.
The declared filename and content type came from the client. Trusting either — as a storage path, as a rendering hint — is how the upload becomes an attack (The Trust Boundary).

Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale

What is actually happening

multipart/form-data is a body format: parts separated by a generated boundary string, each with its own headers, one of which may be a file. Parsing it is a streaming problem — the parser can emit each part as it arrives, or buffer everything and hand you a complete object.
Which of those your framework does decides everything about memory behaviour. Buffering to memory is simple and bounded only by how many uploads arrive at once. Buffering to a temporary file trades memory for disk and adds cleanup you now own. Streaming keeps memory flat and makes the handler harder to write (Request Bodies and Streaming).
A size limit only helps if it is enforced *while* reading, not after. Checking file.size after the framework buffered 200 MB has already spent the memory. Content-Length is a client-supplied claim and may be absent entirely under chunked transfer encoding.
The declared content type is a claim too. The real type is determined by inspecting the bytes — magic numbers at the start of the file — and even that only tells you what it looks like, not that it is safe.
The filename is the most dangerous field on an upload: it may contain path separators, traversal sequences, null bytes, right-to-left override characters, or simply be 4000 characters long. Generate your own storage key and keep the original name as a display label only (Path Traversal).
Storage is a separate write from the database row. Two writes, no shared transaction: the file can land with no row, or a row can point at a file that was never written (The Dual Write Problem).

The path the bytes take

The naive upload has the file crossing the network twice and resident in your process once. Every cost in this lesson follows from that shape: memory, bandwidth, request-slot occupancy, and a size ceiling set by your smallest instance.

It also has a genuine advantage, and it is the reason this path persists: your code sees the bytes. Validation, scanning and rejection all happen before anything is stored, and there is exactly one moment when the file becomes real.

Two transfers — you pay ingress and egress for every byte.
One residency — the file is in your process memory or temp disk.
Two limits — the proxy's and yours; the smaller one wins and only one of them logs.
Two writes — storage and database, with no transaction spanning them.
One advantage — you see the bytes before anyone else does.

UserLLMAgentToolDataDecisionHumanGuardrail

Everything the client tells you about the file is a claim

GENERALType sniffing tells you what the bytes look like, not that they are safe. A valid PNG can carry a payload targeting a specific image decoder, which is why serving from a separate origin still matters (File Upload Security).

Filename, content type, and size are all attacker-controlled fields that arrive looking like metadata. The filename is the worst of them, because it is the one people reach for when naming the stored object — turning a display string into a path.

Turning an upload into a stored object

Client-derived key, declared type

const name = req.file.originalname          // '../../avatars/admin.png'
const type = req.file.mimetype             // 'image/png', because they said so
await storage.put(`tickets/${ticketId}/${name}`, req.file.buffer, {
  contentType: type,
})
await db.attachments.insert({ ticketId, path: name, type })

Generated key, sniffed type, original kept as a label

const detected = await sniffType(req.file.stream)   // magic numbers, not headers
if (!ALLOWED.has(detected)) throw new UnsupportedMedia(detected)

const key = `t/${tenantId}/tickets/${ticketId}/${randomUUID()}`
await storage.put(key, req.file.stream, {
  contentType: detected,
  contentDisposition: `attachment; filename="${sanitize(req.file.originalname)}"`,
})
await db.attachments.insert({
  ticketId, key, type: detected,
  displayName: req.file.originalname,        // data, never a path
})

The first version lets the caller choose where the object lands and what type it will later be served as — path traversal and stored XSS in four lines. The second makes the key unguessable and tenant-scoped, decides the type from the bytes, keeps the user's filename as display data, and sets a disposition so the browser downloads rather than renders it (Serving Files).

Two writes, and what you lose

An upload is a distributed transaction with no coordinator. The object store commits when it commits; your database commits when it commits; a crash between them leaves inconsistency that no rollback repairs.

The useful move is to choose the failure you can clean up. Storage first means orphaned objects — invisible to users, sweepable by a background job, and costing only storage. Database first means rows pointing at nothing, which users see immediately as a broken attachment.

Upload failure modes and responses

Trigger	Symptom	Cause	Response
Ten concurrent large uploads	Process OOM-killed; unrelated requests fail	Framework buffers the full body in memory	Stream to storage; enforce the size limit during the read, not after (Resource Limits).
Upload from a slow mobile connection	Worker pool exhausted, latency rises everywhere	A request slot held for the duration of the transfer	Move the transfer off your process entirely (Presigned URLs), or bound concurrent uploads separately (Bulkheads).
File larger than the proxy limit	413 with no application log line	Rejected upstream of your process	Align proxy and app limits; surface the limit in the API contract so clients can check first (Payload Size: 20KB, 200KB, 5MB).
Crash between storage write and database insert	Object with no row	Two writes, no transaction	Sweep orphans on a schedule; prefer this direction over broken references (The Dual Write Problem).
Client retries a timed-out upload	Two objects, two attachment rows	Non-idempotent upload endpoint	Key the upload by a client-supplied idempotency key or a content hash (Idempotency Keys).
Zip uploaded and expanded to read contents	Disk full	Decompression with no output bound	Cap expanded size and entry count; expand in an isolated worker, never in the request (What Happens After the Bytes Land).
Temp file left after an error	Disk fills over days; no single request is at fault	Cleanup only on the success path	Clean up in a `finally`, and sweep the temp directory on startup.

How to build it

Most important first.

Enforce a size limit at the edge *and* in the application, and enforce it during the read so an oversized body is rejected before it is buffered. The proxy limit and the app limit should be the same number, written down in one place (Configuration: Separating Code From Environment).
Stream to storage rather than buffering, if you keep uploads in the request path at all: read from the request, write to the storage client, never materialize the file in memory.
Generate the storage key yourself — a UUID or content hash, plus a tenant prefix. Never derive it from the client-supplied filename (Tenant Isolation).
Determine the type from the bytes, then check it against an allow-list of types you actually support. A deny-list of dangerous extensions is a list you will never finish.
Write the object first, then the database row, and treat orphaned objects as the acceptable failure — a file with no row is invisible garbage you can sweep; a row with no file is a broken feature (The Transactional Outbox if you need the reverse guarantee).
Decide explicitly whether the bytes need to pass through you at all. For large or numerous files the answer is usually no (Presigned URLs, Choosing an Upload Path).
Do post-upload work asynchronously: scanning, transcoding, thumbnailing and metadata extraction do not belong in the request that carried the bytes (What Happens After the Bytes Land).

What can go wrong

Failure modes

Memory exhaustion from concurrent buffered uploads — the process dies, and the symptom is unrelated requests failing (Unbounded Concurrency).
Temporary files never cleaned up on the error path, filling the disk days later.
Size limit enforced after buffering, so the limit protects storage and not the process.
A proxy 413 or 504 that produces no application log line, making the failure invisible to you and confusing to the user.
Storage write succeeds, database write fails, orphan created; or the reverse, and the ticket shows an attachment that 404s.
A duplicate submission creating two objects and two rows, because upload endpoints are rarely made idempotent (Idempotency in Backends).
A zip bomb or deeply nested archive that is small on the wire and enormous when processed (What Happens After the Bytes Land).
Client disconnects mid-upload, leaving a partial object that a later reader treats as complete.

What can race

Two uploads writing the same key overwrite each other with last-write-wins semantics, silently. Generated unique keys make this impossible; client-influenced keys make it easy.
A reader fetching an object while it is still being written sees a partial or absent object depending on the storage system's visibility semantics (Object Storage).
A client retrying an upload it believes failed, while the original is still in flight, produces two objects and possibly two rows (Idempotency Keys).

Security

If the storage key is derived from the client filename, an attacker gets path traversal: ../../config/app.json as a filename overwrites files outside the intended prefix on a filesystem, or writes into another tenant's prefix in object storage (Path Traversal).
If the declared content type is trusted and files are served from your own origin, an attacker gets stored cross-site scripting: upload an HTML or SVG file, send the link to a victim, and it executes with your origin's cookies (Serving Files).
If there is no enforced size limit, an attacker gets a cheap denial of service: a handful of large concurrent uploads exhausts memory or disk on every instance.
If uploads are unauthenticated or unbounded per user, an attacker gets free storage and bandwidth at your expense, and your bucket becomes a malware distribution host.
If archives are expanded server-side without limits, an attacker gets a decompression bomb — a few kilobytes that expand to terabytes and fill the disk.
If uploaded files are never scanned and are later downloaded by other users, your product becomes a malware delivery channel between your own customers (File Upload Security).
If the upload endpoint is not rate limited, an attacker gets both the DoS and the storage-cost attack for the price of one script (Rate Limiting).

Misreads

"The framework handles multipart, so uploads are handled." It handles parsing. Memory behaviour, limits, key generation, type checking and the second write are all yours.
"We check the extension." Extension and declared content type are both client-supplied strings. Only the bytes are evidence, and even then only weakly.
"Signed URLs are always better." They remove your ability to validate before storage, which is exactly what some products need (Choosing an Upload Path).
"The file is in the database transaction." It is not. Object storage has no participation in your database transaction, ever (Transactions from Application Code).
"Small files, so it does not matter." It matters when a client sends a large one, which is a thing clients do — deliberately, once they notice there is no limit.

Operating it

How you see it in production

Track upload duration and size distributions separately from other endpoints. Upload latency is dominated by the client's uplink, so mixing it into API percentiles makes both meaningless (Percentiles: Which One, and How Many Users Is That?).
Alert on process memory relative to concurrent in-flight uploads. A rising ratio is the signal before an OOM kill (The Metrics a Backend Must Emit).
Count rejections by reason — too large, wrong type, failed magic-number check — so a client bug is distinguishable from probing.
Reconcile periodically: objects with no database row, and rows with no object. Both counts should be near zero and neither is visible from a request log (Health Checks: Startup, Readiness, Liveness will not catch this).
Log the generated storage key, never the client-supplied filename verbatim — it is untrusted content and it lands in your log pipeline (Secrets in Logs).

What changes at 10x and 100x

At 10x uploads, the bandwidth through your instances is the constraint before CPU is. Instances sized for JSON handling are the wrong shape for file traffic.
At 100x, keeping bytes in the request path stops being viable at any instance count: you are paying for ingress, egress and compute to move data that could go directly to storage (Presigned URLs).
Nothing changes for small files at low rates, and that is the point — a 100 KB avatar upload through your API is fine forever, and rewriting it to use signed URLs buys nothing (Choosing an Upload Path).
Very large files need resumable or multipart uploads regardless of path, because a single HTTP request that must complete is a bad bet over a mobile network.

What this costs

Routing bytes through the backend gives you synchronous validation, virus scanning before storage, and one place that knows the file exists. It costs memory, bandwidth, request slots and a hard ceiling on file size.
Streaming instead of buffering keeps memory flat and makes the handler harder to write, harder to test, and harder to roll back on a mid-stream failure.
Writing storage before the database means orphaned objects; writing the database first means broken references. There is no ordering that gives you both, only a choice of which garbage you prefer (The Dual Write Problem).
Strict type allow-lists will reject files users legitimately want to attach, and every one of those becomes a support ticket.

Where this applies

Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.

GENERALThe trust, sizing and dual-write concerns apply to every stack.
FRAMEWORK-SPECIFICDefault buffering behaviour differs sharply and decides your memory profile: Express with multer defaults to memory storage unless you configure a disk or stream destination; Django writes anything over a configurable threshold to a temporary file; Rails ActiveStorage stages to a temp file then uploads. Read your own framework's default before assuming which one you have.
RUNTIME-SPECIFICOn Node a slow upload consumes an in-flight request but not the loop thread, so it degrades memory before latency; on a thread-per-request or pre-fork model (a synchronous Python or Ruby app server) a slow upload occupies a whole worker, so a small number of slow clients exhausts concurrency long before memory (Backend Runtime Models).

Where the depth lives

This domain teaches the application-side mechanism and hands the rest off.

API Designfile-upload-apis payload-size large-requests

Securityfile-upload-security path-traversal input-validation

Performancepercentiles

Cloudobject-storage