Video Processing
Case Study: Video Upload and Transcoding
Users upload video; the platform produces several playback renditions and serves them worldwide. Structurally this is
cs-worker-pipeline with much larger payloads and much more expensive work — and that changes the design in ways the smaller pipeline never had to consider. The workers are the entire bill. Every architectural decision here is really a cost decision wearing an engineering costume, and treating it as anything else is how a video product becomes unprofitable at exactly the moment it becomes popular.Requirements
- Accept video uploads up to several gigabytes over unreliable consumer connections.
- Produce multiple renditions per source — several resolutions plus a thumbnail — and make them available for playback.
- Serve playback to a global audience with acceptable startup time.
- Absorb a 20x burst without a queue that takes days to drain.
- Keep unit economics predictable: cost per minute of source video must be a number the business can quote.
Deliberately not requirements
Half of a design is what it refuses to do. These are the refusals.
Out of scope, on purpose
- No real-time or live streaming — that is a fundamentally different architecture with a latency budget this design does not have.
- No editing, no per-viewer watermarking, no DRM at this stage.
- No guarantee of transcode completion time beyond a stated, generous target.
How the design got here
In order. Each stage leads with the problem that forced it.
Stage 1
Upload and transcode on the API instance
Forced by
The requirement, at fifty videos a day. The API accepted an upload, ran the encoder as a subprocess, wrote the outputs beside the source, and served them from the same machine. One deployable, no queue, no storage service, no CDN.
Everything on the box that also serves the API. It works, briefly.PROVIDER-NEUTRAL
Uploaderpublic
Load balancerpublic
API + encoder VMprivate— accepts the upload, runs the encoder inline, serves playback from local disk
Local volumeprivate— sources and renditions; fills predictably
Uploader→Load balancer· POST videocrosses boundary
Load balancer→API + encoder VM
API + encoder VM→Local volume· write source and renditions
Uploader→Load balancer· GET playbackcrosses boundary
| Decision | Reason | Alternative | Trade-off |
|---|---|---|---|
| One machine, no pipeline. | At fifty videos a day the encoder is idle most of the time and the whole feature is a subprocess call. Building a distributed pipeline first would have been infrastructure for a product that did not yet exist. | A managed transcoding service, which removes almost all of this case study and charges per minute of output — genuinely the right answer for many teams, and the alternative to keep in mind at every later stage. | The ceiling is close and the failure at the ceiling is total: a single large upload can fill the disk, exhaust memory, and starve the API of CPU for everyone at once. |
Stage 2
Direct uploads to object storage, events to a queue
Forced by
A 4 GB upload streamed through an API instance filled its disk and pushed it into OOM; the request also had to survive twelve minutes on a mobile connection, and any interruption meant starting over. Separately, an autoscaling event replaced an instance and silently deleted every rendition it held.
Bytes stop touching the application. Storage becomes the pipeline's entry point.PROVIDER-NEUTRAL
Uploaderpublic
APIprivate— issues a multipart signed upload URL and records metadata — never handles the bytes
Source bucketprivate— resumable multipart uploads; a lifecycle rule for abandoned parts
Transcode queueprivate— one message per rendition, not per video — the unit of retry and of parallelism
Metadata DBprivate— video and rendition status
Uploader→API· request upload URLcrosses boundary
Uploader→Source bucket· multipart PUTcrosses boundary
Source bucket→Transcode queue· object-created event
API→Metadata DB· create video record
| Decision | Reason | Alternative | Trade-off |
|---|---|---|---|
| Resumable multipart uploads directly to object storage. | A multi-gigabyte upload over a consumer connection will be interrupted. Multipart makes interruption cost one part rather than the whole file, and it removes the upload entirely from your compute. | Proxy uploads through the API with a resumable protocol implemented yourself, which keeps validation inline and is a large amount of code to write badly. | Abandoned multipart uploads accumulate as invisible, billable storage — parts that belong to no object and appear in no listing. A lifecycle rule to abort them is not optional; it is the difference between a storage bill and a mystery. |
| The storage event, not the API, starts the pipeline. | The authoritative moment is "the object exists", and only storage knows it. Enqueueing from the API means enqueueing work for a file that might never finish uploading. | The client calls a "finish" endpoint, which is simpler to trace and depends on a client that may crash, lose signal, or lie. | Event delivery is at-least-once and occasionally delayed, so the pipeline must be idempotent per rendition and needs a sweeper for objects that exist but were never processed. |
Stage 3
A dedicated transcoding pool
Forced by
Transcoding pegged the API instances at 100% CPU for minutes at a time. Interactive requests queued behind encoder processes, p99 latency went from 80ms to eleven seconds, and one user uploading a long video made the product unusable for everybody else.
Two pools with opposite requirements, scaled and sized independently.PROVIDER-NEUTRAL
API poolprivate— small instances, latency-sensitive, scales on request rate
Transcode queueprivate
Transcoding poolprivate— CPU-optimized instances, throughput-sensitive, scales on queue age; one job saturates one instance by design
Source bucketprivate
Output bucketprivate
Metadata DBprivate
Transcode queue→Transcoding pool· receive rendition job
Transcoding pool→Source bucket· stream source
Transcoding pool→Output bucket· write rendition
Transcoding pool→Metadata DB· update rendition status
| Decision | Reason | Alternative | Trade-off |
|---|---|---|---|
| A separate, CPU-optimized pool for transcoding. | The two workloads want opposite machines. The API wants many small instances with headroom for bursts; the encoder wants the most cores per euro and will use all of them. Mixing them means sizing for neither. | One pool with resource limits per container, which is cheaper to operate and caps the encoder at a fraction of the machine — turning a CPU-bound job into a slower CPU-bound job. | Two instance families, two scaling policies, two capacity conversations. And an idle transcoding pool is far more expensive per hour than an idle API pool, which is the problem the next stage exists to solve. |
| One message per rendition, and one rendition per instance at a time. | The encoder already parallelizes across all cores, so running two jobs on one machine roughly doubles each one's wall-clock time without improving throughput. It also makes the unit of retry small and the unit of cost easy to attribute. | Several concurrent jobs per instance, which improves utilization when jobs are small and makes per-job latency unpredictable. | Utilization within an instance is never quite 100%, and the instance is held for the whole job even during its I/O phases. You are trading a little efficiency for predictability and simple accounting. |
Stage 4
Elastic and interruptible capacity
Forced by
The pool was sized for the daily peak and sat below 5% utilization overnight, while transcoding was already the largest line on the invoice — the company was paying peak prices for idle CPU. In the same quarter a customer's 900-video migration still took fourteen hours, because the peak-sized pool was also the ceiling.
Capacity follows the queue, and most of it is bought at interruptible prices.PROVIDER-NEUTRAL
Transcode queueprivate— oldest-message age drives the policy
Baseline pool (on-demand)private— a small floor of reliable capacity for latency-sensitive short jobs
Interruptible poolprivate— the bulk of capacity, at a large discount, reclaimable with little notice
Checkpoint prefixprivate— completed output segments, so an interrupted job resumes instead of restarting
Output bucketprivate
Transcode queue→Baseline pool (on-demand)· short and urgent jobs
Transcode queue→Interruptible pool· bulk jobs
Interruptible pool→Checkpoint prefix· write segments as they finish
Checkpoint prefix→Interruptible pool· resume from last segment
Interruptible pool→Output bucket· assemble final rendition
| Decision | Reason | Alternative | Trade-off |
|---|---|---|---|
| Most transcoding runs on interruptible capacity. | The workload is batch, restartable and latency-tolerant, which is exactly the profile interruptible capacity is priced for. On a bill dominated by compute, this is the single largest lever available. | On-demand capacity throughout, which is simpler and predictable, and costs multiples for identical work. | Instances are reclaimed with little warning, so every job must be resumable and every output must be written idempotently. Capacity is also not guaranteed: during a regional shortage your throughput can drop to the baseline pool, which is why the baseline exists. |
| Segment-and-checkpoint long transcodes. | A forty-minute job interrupted at 90% and restarted from zero costs you 36 minutes of compute *and* the queue latency again. Segmenting turns the cost of an interruption into one segment. | Restart interrupted jobs from the beginning, which needs no extra code and makes interruptible capacity a false economy for long videos. | Segmenting, tracking and assembling adds real complexity, extra storage for intermediates, and a new failure mode where segments exist but assembly never runs. Worth it above a duration threshold, actively harmful below one. |
| Scale on queue age with a ceiling tied to a spend budget. | Compute here is unbounded by nature — a bad batch could scale to hundreds of instances and produce an invoice nobody approved. The ceiling turns an unbounded financial risk into a bounded latency cost. | Unbounded scaling for the best possible drain time, which is the correct choice only if someone has explicitly accepted the maximum possible bill. | A large burst takes longer than it technically could, and during a genuine spike the queue-age alert fires while the system is behaving exactly as configured. That alert needs to say so, or it will be misdiagnosed every time. |
Stage 5
CDN delivery and storage lifecycle
Forced by
Playback pulled every segment directly from the origin bucket. One popular video generated more egress in a single day than a week of transcoding compute had cost, and viewers outside the origin region buffered constantly. Meanwhile the source bucket had grown past the point where anyone could explain what was in it: every original ever uploaded, kept forever, at hot-tier prices.
Delivery moves to the edge, and stored bytes acquire a lifecycle.PROVIDER-NEUTRAL
Viewerpublic
CDNpublic— caches segments and manifests at the edge; signed URLs enforce access
Output bucket (hot)private— renditions and manifests; the CDN is the only reader
Source bucketprivate— lifecycle: hot for 30 days, then cold, then archive
Archive tierprivate— originals kept for re-encoding; retrieval is slow and separately charged
Viewer→CDN· HTTPS playbackcrosses boundary
CDN→Output bucket (hot)· origin fetch on miss
Source bucket→Archive tier· lifecycle transition
| Decision | Reason | Alternative | Trade-off |
|---|---|---|---|
| All playback goes through the CDN; the output bucket has exactly one reader. | Video is the highest-volume egress a product can have, and edge delivery both reduces the per-gigabyte rate and removes repeat traffic from the origin. It is a latency improvement and a cost improvement at the same time, which is rare. | Serve directly from object storage, which is one fewer component and pays origin egress rates for every viewer, every time. | Cache invalidation on a re-encode, signed-URL expiry that must outlive a viewing session but not a shared link, and a hard dependency on the CDN for the product's core function. |
| Keep originals, but move them down the storage tiers automatically. | Originals are the only thing you cannot regenerate, and re-encoding for a new device profile in two years needs them. But they are read almost never after the first month, which is the exact profile cold storage is priced for. | Delete originals after successful transcoding, which is cheapest and means a future format change silently costs you the entire back catalogue. | Retrieval from an archive tier is slow and separately billed, and there are minimum storage durations — moving an object down and pulling it back quickly can cost more than never moving it. Lifecycle rules are a bet on access patterns. |
| Choose the rendition ladder deliberately, and revisit it. | Every rendition multiplies transcoding compute *and* stored bytes. A ladder of five renditions costs roughly five times a ladder of one, forever, for videos that may never be watched. | Encode on first playback rather than on upload, which eliminates cost for unwatched videos and adds latency to the first viewer — an excellent trade for long-tail libraries. | Fewer renditions means worse playback on slow connections, which is a product-quality decision the infrastructure team should not be making alone. |
Stage 6
Poison inputs and per-job cost limits
Forced by
One corrupt source file crashed the encoder on every attempt, burning a full instance-hour each time and retrying indefinitely. A separate malformed file caused the encoder to run without terminating: a single job consumed capacity for eleven hours before anyone noticed a cost anomaly. Both were retry loops that no depth or age metric surfaced.
Bounded attempts, bounded duration, bounded spend per job.PROVIDER-NEUTRAL
Transcode queueprivate— bounded receive count
Transcoding workerprivate— hard wall-clock timeout per job, sized from source duration; probe the file before encoding it
Dead-letter queueprivate— unprocessable sources, with the encoder's own error preserved
Metadata DBprivate— per-rendition status, attempts, and measured compute seconds — the unit-economics record
Cost anomaly alertprivate— alerts on compute-seconds per source minute drifting from its baseline
Transcode queue→Transcoding worker
Transcode queue→Dead-letter queue· after bounded attempts
Transcoding worker→Metadata DB· record attempts and compute seconds
Metadata DB→Cost anomaly alert· unit-cost baseline
| Decision | Reason | Alternative | Trade-off |
|---|---|---|---|
| A hard wall-clock timeout per job, derived from source duration. | Transcoding time is roughly proportional to source length, so "this job may run for at most N times real time" is both enforceable and diagnostic. It converts an unbounded runaway into a bounded, visible failure. | No timeout, trusting the encoder to finish, which is how one file consumed eleven hours of a paid instance. | A legitimately hard encode can be killed by a timeout that was tuned for the common case, so the multiplier needs headroom and the killed jobs need somewhere to be seen. |
| Probe the source before committing an instance to it. | A cheap metadata probe catches corrupt, truncated and absurd inputs — a 40-hour file, a codec you do not support — in seconds instead of after an hour of wasted compute. | Let the encoder discover the problem, which is simpler and pays full price for every rejection. | A second tool in the pipeline with its own failure modes, and a probe that occasionally rejects a file the encoder would have handled fine. |
| Track compute-seconds per source minute as a first-class metric. | On a compute-dominated bill, unit cost *is* the health metric. A change in that ratio detects a bad encoder setting, a retry storm or an inefficient ladder long before the monthly invoice does. | Watch the total bill, which is a lagging monthly signal that mixes growth with regression and cannot distinguish them. | Instrumentation and attribution work that produces no user-visible feature, and a baseline that must be re-established after every deliberate change to the ladder or the encoder. |
What would break this
Every design has a load, a failure or an organization size at which it stops being the right one.
Breaking points
- Live streaming. Ingest, low-latency packaging and continuous delivery are a different architecture; nothing in this batch pipeline transfers except the CDN.
- Per-viewer output — dynamic watermarking, personalized ad insertion — which destroys the cache-once, serve-many economics that make delivery affordable.
- DRM and content protection, which adds a license server, key rotation and a packaging step, and forces most of the delivery decisions to be revisited.
- A catalogue large enough that storage overtakes compute. Past that point the design question flips from "how do we transcode cheaply" to "what are we allowed to delete", and lifecycle policy becomes the main lever.
- GPU-accelerated encoding. It is faster per job and priced differently per hour; whether it is cheaper depends entirely on utilization, and it is a right-sizing exercise, not an upgrade.
- Interruptible capacity becoming scarce in your region. The design silently degrades to baseline throughput, so the baseline pool size is a real availability decision, not a rounding detail.
Cost shape
Drivers and relative weights. Never a price.
A compute-dominated bill — until the moment delivery overtakes it. Shapes and relative weights only; never a price.COST-VARIES
Transcoding compute spiky
driven by instance-hours ≈ minutes of source × renditions × encoder speed factor · The workers are the bill. Every lever that matters — interruptible capacity, ladder size, probe-before-encode, per-job timeouts — points at this line.
CDN delivery · the surprisespiky
driven by gigabytes streamed to viewers = watch-minutes × bitrate · Driven by viewers, not by uploads, so it is entirely decoupled from the rest of the pipeline. One viral video can make this line exceed all compute for the month.
Output storage usage
driven by gigabytes of renditions × the ladder multiplier, retained forever · Grows monotonically with the catalogue and multiplies by the number of renditions. The ladder decision is a storage decision as much as a compute one.
Source storage usage
driven by gigabytes of originals, tiered by age · Cheap per gigabyte once lifecycle rules move it down, and un-deletable in practice if you ever want to re-encode.
Abandoned multipart parts · the surpriseusage
driven by incomplete uploads never aborted · Invisible in a bucket listing and fully billable. The purest example of a cloud cost that exists only because a lifecycle rule was never written.
Queue, API, database fixed
driven by requests and instance-hours for the coordination tier · Nearly irrelevant here, which is itself the lesson: optimizing this tier on a video platform is optimizing the wrong thing.
Origin egress to the CDN usage
driven by gigabytes fetched on cache miss · Small when the cache hit ratio is high, and a direct multiplier of the delivery bill when it is not — which is why a re-encode that invalidates everything is a costly operation.
Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.
Lessons behind this design