File Upload APIs: Authorize, Upload Directly, Confirm
The API that handles JSON should not be the pipe for a 3GB video. Create an upload resource, hand the client a signed URL to object storage, confirm completion, then process asynchronously — a four-step contract that keeps the API small, the bytes off your workers, and retries safe.
Frame the contract
API design starts with a consumer, a design question and a guarantee — never with a URL.
Why the API must not be the pipe
POST /documents with a multipart body works for a 200KB PDF and fails for a 3GB video in every way at once. The API worker holds the connection for the whole transfer — minutes, on a phone — and cannot serve anything else; the proxy buffers the body (Large Requests and Documented Limits limits apply, and they should be small); a network blip at 95% loses everything with no resume; the bytes then get copied again from the worker to storage, doubling the transfer. The API's job is to *decide* about the upload; the bytes belong to a system built to receive bytes.
The pattern is three API calls around a direct transfer. Create: POST /uploads with metadata (filename, size, content type, purpose) — the API authorizes (Authorization Design in the Contract: may this user attach a file to this project?), validates the declared size and type against limits, records an upload resource in state pending, and returns a signed URL for object storage that expires in minutes. Upload: the client PUTs the bytes to that URL directly; the API is not involved. Complete: POST /uploads/{id}/complete — the API verifies the object exists with the expected size (and optionally checksum), moves the upload to uploaded, and starts processing.
The signed URL is the load-bearing security element: it is scoped to one object key, one method, one size range, one content type, and one time window, so possession grants exactly one upload and nothing else. The security domain's File Upload Security lesson covers the content-side checks (type sniffing, malware scanning, never serving user bytes from the API origin); the contract's part is that none of those checks happen on the API worker during the transfer.
Retries, resumption and the states in between
Every step fails independently, and the contract says how each recovers. Create is idempotent with an Idempotency Keys: The Mechanism key, so a lost response does not mint two upload resources. The direct upload retries against the same signed URL until it expires; for large files, storage-native multipart or resumable protocols let the client upload in parts and resume from the last acknowledged byte — the API can expose this by returning a set of part URLs, or a resumable session URL, from the create step. Complete is idempotent by construction: completing an already-completed upload returns the current state.
The upload resource is a state machine (Resources Have State Machines): pending → uploaded → processing → ready, with failed reachable from processing (corrupt file, scan rejection) and expired from pending (the client never finished). Each state is a contract clause the client can build UI on: show progress in pending, a spinner in processing, the file in ready, and the specific error (FILE_REJECTED_MALWARE, UNSUPPORTED_FORMAT) in failed. Processing is the The Async Job Pattern applied to a file — long-running by nature, and reported the same way (How the Client Learns the Job Finished).
Expiry is the cleanup contract. Pending uploads whose URL expired are garbage-collected after a stated window; storage objects that were uploaded but never completed are deleted after a stated window; the client learns both from the docs and from 410 Gone on the upload resource. Without stated expiry the storage bucket fills with half-uploads nobody can attribute.
POST /projects/proj_18a2/uploads
Idempotency-Key: 5c0d…
{ "filename": "footage.mov", "size_bytes": 3221225472, "content_type": "video/quicktime", "purpose": "source_video" }201 Created
{
"id": "upl_c41",
"status": "pending",
"upload": {
"method": "PUT",
"url": "https://storage.example/bucket/proj_18a2/upl_c41?X-Sig=…",
"headers": { "Content-Type": "video/quicktime" },
"expires_at": "2026-08-25T10:30:00Z",
"max_bytes": 5368709120
},
"complete_url": "/uploads/upl_c41/complete"
}
# Client PUTs 3 GB to storage, then POST /uploads/upl_c41/complete → 202 { "status": "processing" }Metadata, limits and the download side
Metadata travels with the create call, not with the bytes: filename, declared size and type, the owning resource, user-supplied tags. The API validates it — size against the per-purpose limit (a 5GB cap for source video, 10MB for avatars), type against an allowlist, ownership against authorization — before issuing a URL. After completion, the server-observed facts (actual size, sniffed type, checksum, dimensions or duration from processing) override the declared ones in the response; the declared values were the client's claim, the observed values are the contract's truth.
The download side mirrors the pattern. GET /uploads/{id} returns metadata plus a short-lived signed download URL, or redirects to one; the API does not stream bytes. Public files may be served through a CDN (CDN Architecture); private ones through expiring URLs. Either way, the API response model (Response Contracts Are Not Database Rows) exposes the file's identity and state, never its storage path.
The cost of the pattern is honesty about complexity: three calls where naive design had one, a signed-URL mechanism to run, a cleanup job for orphans, and a client SDK that hides the choreography (SDK Design: The Contract's User Interface — uploads.create(file) should do all of it). The payoff is an API that stays fast under a thousand concurrent uploads because none of them touches it, and a client that can survive a dropped connection at 95%.
- Three calls: create (authorize, metadata, signed URL), direct upload to storage, complete (verify, start processing).
- Signed URLs scoped to one key, method, size, type and time window.
- State machine: pending → uploaded → processing → ready | failed | expired, each a clause clients build on.
- Resumable/multipart for large files; every step idempotent.
- Observed facts override declared; downloads via signed URL or CDN, never streamed through the API.
Key points
- The API authorizes and records uploads; object storage receives the bytes. Proxying large files through the API costs workers, doubles transfers and breaks resumption.
- Create → direct PUT → complete → async processing is the contract; each step is independently retryable and the upload resource is a state machine.
- A signed URL is scoped to one object, method, size, type and time window — possession grants exactly one upload.
- Declared metadata is validated up front; server-observed facts (actual size, sniffed type, checksum) become the truth after processing.
- Stated expiry for pending uploads and orphaned objects is part of the contract; the SDK hides the choreography from consumers.
Follow the failure
How the contract fails or gets misused, hop by hop — and what it costs when it completes.
- 1Team → API:
POST /documentswith a multipart body; fine for PDFs in the demo. - 2Product → video: users upload 3GB files; each holds an API worker for minutes; the proxy limit is raised to 5GB to "fix" the 413s.
- 3Mobile → upload: connection drops at 95%; the whole transfer restarts; the user gives up.
- 4Worker → storage: bytes are copied from the worker to the bucket after receipt, doubling transfer time and memory.
- 5On-call → incident: twenty concurrent uploads exhaust the worker pool; every JSON request on the API times out.
- API capacity is consumed by byte transfer instead of decisions; unrelated endpoints degrade under upload load.
- No resumption: every network failure costs the entire transfer, which on mobile means most large uploads never complete.
- Untracked bytes: without an upload resource and completion step, the system cannot tell an in-progress upload from an abandoned one.
Design, observe, evolve
A contract decision is incomplete until you know how you would notice it failing and how it changes later.
- • Model uploads as resources with a state machine; create authorizes and returns a scoped, expiring signed URL; complete verifies and triggers processing.
- • Send bytes directly to object storage with multipart or resumable protocols for large files; the API never streams them in either direction.
- • Make every step idempotent (create via idempotency key, complete by construction) and state expiry windows for pending uploads and orphaned objects.
- • Validate declared metadata against per-purpose limits before issuing a URL; record observed facts after processing.
- • Ship the choreography in the SDK so consumers call one method and get progress, resumption and status for free.
- • Upload funnel by state — created, uploaded, completed, ready — per client type shows where transfers die (a pending-to-uploaded cliff on mobile means resumption is missing).
- • Signed-URL issuance vs. completion counts reveal orphan rates and whether expiry windows are right.
- • API worker time spent on upload endpoints should be milliseconds; anything else means bytes are flowing through the API.
- • Adding resumable/multipart support is additive: the create response gains a session or part URLs while the single-PUT form keeps working.
- • New processing outputs (thumbnails, transcripts) are additive fields on the ready state; new failure codes are additive to the failed state.
- • Changing storage providers is invisible to consumers because the contract exposes upload ids and signed URLs, never bucket paths.
- • Three calls and a signed-URL mechanism are more moving parts than one multipart POST, and the SDK exists to hide them.
- • Direct upload means the API cannot inspect bytes in flight; validation moves to the processing step, after the bytes are already stored.
- • Orphan cleanup and expiry are operational jobs the simple design never needed.