Case Study: File Upload API
Document storage for a product where users attach files up to 5 GB, which are scanned and processed before they become usable.
The defining decision in an upload API is what your servers should *not* do: proxy the bytes. A 2 GB file streamed through your API ties up a connection and a worker for minutes, doubles your bandwidth bill, and turns every deploy into a dropped upload. The design that avoids it splits the flow into three contracts: create an upload (your API, small request), send the bytes (signed URL, direct to storage — File Upload APIs: Authorize, Upload Directly, Confirm), complete and process (your API again, returning 202 because scanning takes time — Long-Running Operations: 202 and the Job Resource). Everything else in the study — resumability, checksums, expiry — falls out of taking each of those three steps seriously.
Consumers
Drag-and-drop with a progress bar and the ability to resume a 2 GB upload after a laptop sleeps mid-transfer.
Uploads photos over flaky cellular links — small parts, per-part retry, and no penalty for finishing a multi-part upload an hour late.
Bulk-import documents via API keys; need deterministic completion status and a way to attach metadata (source system, external id) queryable later.
Requirements
- • Files up to 5 GB; the API tier must never carry the payload bytes.
- • Uploads survive interruption: a network drop loses at most one part, never the whole transfer.
- • A file is not visible or downloadable until it has passed virus scanning and processing (thumbnails, text extraction).
- • The client can attach metadata (filename, content type, custom key/values) and must find out authoritatively when processing finished or failed.
- • Integrity is verified: what storage received is what the client sent, checksum-proven.
- • Abandoned uploads don't accumulate cost forever.
Resources
The transfer *session* — a short-lived resource with its own lifecycle (`created` → `uploading` → `processing` → `completed` | `failed` | `expired`). Making the session explicit is what makes resumability, expiry, and progress addressable.
One chunk of a multipart transfer (32 MB each). Parts are the retry unit: a drop at 96% costs one part, not 5 GB. Each part has its own signed URL and checksum.
The durable artifact that exists only after processing succeeds. Separating File from Upload keeps half-transferred bytes out of every listing and means `GET /files/{id}` never returns something you can't actually download.
Exposed as the upload's `processing` sub-state with per-step detail (scan, thumbnail, extraction) rather than a separate top-level resource — visible enough to debug, not so prominent that clients couple to pipeline internals.
Operations
| Operation | Purpose | Design notes |
|---|---|---|
| POST /uploads | Create an upload session; declare size, content type, checksum algorithm, metadata. | Returns 201 with signed part URLs and their expiry. Declaring size up front lets the API reject a 12 GB file *before* any bytes move (413) and lets storage enforce exact-size policies on the signed URLs. Idempotency-keyed so a timed-out create doesn't strand an orphan session. |
| PUT {signed storage URL} | Send one part's bytes directly to object storage. | Not your API — and documented anyway, because it's the step clients get wrong. Each URL is scoped to one part, one content length, one time window: possession of a URL is the entire authorization, so the URL must be narrow (Authentication in the Contract). |
| GET /uploads/{id} | Session status: which parts are received, processing progress. | The resume path: a client that lost local state asks the server which parts landed and re-sends only the gaps — server state is the truth, client bookkeeping is a cache. |
| POST /uploads/{id}/parts/{n}/refresh | Re-issue an expired signed URL for one part. | URLs expire in 1 hour for security; big files legitimately outlive that. Refresh keeps expiry short without making slow uploads impossible. |
| POST /uploads/{id}/complete | Declare all parts sent; submit part checksums; start processing. | Returns `202 Accepted`, not 200: scanning and thumbnailing take seconds to minutes, and holding the connection open would just re-create the proxying problem one step later. The response carries the upload in processing state and a Retry-After polling hint. Checksum mismatch is a 409 here — the one moment integrity can still be fixed by re-sending a part. |
| GET /files/{id} | Fetch file metadata and a short-lived download URL. | Download is also direct-from-storage via signed URL — the no-proxy rule applies in both directions. |
| GET /files | List completed files, cursor-paginated, filterable by metadata keys. | Only completed files appear; in-flight uploads live under /uploads. Two collections, two lifecycles, no status filter every client must remember to apply. |
| DELETE /uploads/{id} | Abort an in-progress upload and free storage. | Also happens automatically: sessions untouched for 24h expire server-side, which is the answer to "abandoned uploads cost money forever". |
Error contract
| Code | Status | When | Retryable |
|---|---|---|---|
| FILE_TOO_LARGE | 413 | Declared size exceeds the 5 GB limit — rejected at session creation, before any bytes move. | no |
| UNSUPPORTED_MEDIA_TYPE | 415 | Content type not on the allowlist for this workspace (e.g. executables). | no |
| UPLOAD_INCOMPLETE | 409 | `complete` called while parts are missing. `details.missing_parts` lists exactly which — the response *is* the recovery instruction. | no |
| CHECKSUM_MISMATCH | 409 | A part's checksum doesn't match what storage received — corruption in transit. Re-send that part and complete again. | no |
| UPLOAD_EXPIRED | 410 | Any operation on a session past its 24h lifetime. `410` tells the client to start a new session, not to retry this one. | no |
| PROCESSING_FAILED | 200 | Not an HTTP error at all: `GET /uploads/{id}` returns `200` with `status: "failed"` and a `failure` object (`virus_detected`, `corrupt_file`). The transfer succeeded; the *outcome* was negative — states, not statuses ([[async-job-pattern]]). | no |
Decision log
Decision → reason → alternative → trade-off. The alternative is part of the record.
file.ready webhook was added as an *optimization* for server-side integrations — polling remains the contract's floor./files is downloadable, period. Merging them forces every consumer to filter by status forever, and the one who forgets ships a UI full of broken attachments.File resource with status: uploading | processing | ready.409 at upload-time, when the client still has the original.How it evolves
- • Resumable single-stream protocol (tus-style) for small mobile files arrives as an *alternative* transfer mode declared at session creation — the Upload resource and completion contract are unchanged, so consumers of upload status never know which wire protocol carried the bytes.
- • Image variants on demand (
?width=400) extendGET /files/{id}with query parameters; the unparameterized URL keeps returning the original, so existing links never change meaning. - • Completion webhooks (
file.ready,file.failed) are added on the existing event vocabulary for integrations that don't want to poll — additive, with polling still documented as the fallback truth (How the Client Learns the Job Finished). - • Retention policies: files gain an optional
expires_at;GET /files/{id}on an expired file returns410with a documented error code, announced through a deprecation-style comms window because it changes a previously-permanent promise (Deprecation as a Process, Not a Label).