Model Compression
Quantization, pruning and distillation are three different bargains: fewer bits per weight, fewer weights, or a smaller model taught by the larger one. They trade quality, latency, cost and engineering effort differently, and can be combined.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The model is too slow or too big for where it has to run. Which way of making it smaller fits this constraint, and what does each cost in quality and effort?
A speech-intent model runs on a cloud GPU and the product now needs it on a mid-range phone, offline, answering within a fraction of a second. The team has one model that works and a list of techniques — quantize, prune, distil — and no idea which to try, in what order, or how much quality they will lose.
Quantize to eight bits and see. If it is still too big, prune half the weights. If it is still too slow, train a smaller model from scratch. Each step is a known technique with a library call.
Eight-bit quantization halves or quarters the memory and does little for latency on a CPU whose bottleneck is the number of multiply-adds, not the bytes; the phone is compute-bound where the cloud GPU was memory-bound.
- Eight-bit quantization halves or quarters the memory and does little for latency on a CPU whose bottleneck is the number of multiply-adds, not the bytes; the phone is compute-bound where the cloud GPU was memory-bound.
- Unstructured pruning zeroes half the weights and the file compresses nicely — and runs at exactly the same speed, because the mobile kernel does dense arithmetic and multiplies the zeros (Pruning & Distillation).
- The smaller model trained from scratch on the labelled data is markedly worse on rare intents; it never had enough examples of them. The technique that would have helped — distillation from the cloud model on the unlabelled corpus — was the one tried last.
- Every step was evaluated on the aggregate, and the accent slices that the product team cares about were not checked until a beta tester complained.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The surrounding model maps a spoken phrase to one of a few hundred intents; the compression's target is a model that fits the phone's memory and latency budget with quality that the product accepts on every slice that matters — accents, noisy environments, rare intents.
- The decision is a plan: which technique first, how far to push it, and when to stop — with the quality budget spent deliberately.
- The cloud model is a mid-sized transformer in full precision; the phone budget is a small fraction of its memory and a hard per-utterance latency on a mobile CPU with limited vector units.
- Training data exists for the original model, plus a large unlabelled corpus of utterances that the cloud model could label.
- The slice evaluation from the cloud model's launch is available and is the yardstick for every compressed candidate.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- The three techniques attack different terms in the cost. Quantization keeps every weight and shrinks its representation — it reduces bytes and, on hardware with fast low-precision units, arithmetic time. Pruning removes weights — it reduces arithmetic only if the removal is structured so that the kernel can skip whole blocks. Distillation replaces the model with a smaller one — fewer layers, narrower — trained to reproduce the large model's outputs rather than only the hard labels (Pruning & Distillation).
- Which term binds depends on the target device. On a memory-bound GPU pass, quantization is nearly free speed. On a compute-bound mobile CPU, only fewer multiply-adds help — structured pruning or a smaller architecture — and quantization mostly helps memory and cache behaviour.
- The techniques compose. A distilled model can be quantized; a pruned model can be distilled into. Each step spends part of a quality budget, and the budget is measured per slice, because each technique loses quality in a different place: quantization on outlier-sensitive rare cases, pruning on whatever the pruning criterion undervalues, distillation on whatever the teacher was uncertain about (Quantization).
- Compression is a modelling change and inherits every rule of one: the compressed artifact is a new model with its own evaluation, promotion, lineage and rollback (Promotion Is a Checklist, Not a Score).
Three different bargains
The techniques are usually listed together and are not alternatives. Quantization is about representation: same network, fewer bits. Pruning is about connectivity: same architecture, fewer weights, and only faster if the removal is structured. Distillation is about size: a different, smaller network, taught to imitate the big one.
They line up against different bottlenecks, which is why the plan starts from the device and not from the list.
| Option | Quality | Latency | Cost | Operational | Data needed | Note |
|---|---|---|---|---|---|---|
| Post-training INT8 quantization | Minutes of work; large memory reduction; small latency gain on a compute-bound CPU; quality risk concentrated on outlier-sensitive rare inputs. | |||||
| Structured pruning + fine-tune | Removes whole channels or heads so dense kernels run faster; needs a fine-tuning pass to recover; the criterion decides which slices suffer. | |||||
| Unstructured pruning | High sparsity at little aggregate quality cost; almost no speed-up on hardware that does dense arithmetic; the file compresses, the runtime does not. | |||||
| Distillation to a small student | The fastest result at a given quality; a training run and a teacher pass over a large corpus; rare slices gain signal from the teacher's soft outputs. | |||||
| Distilled student, then INT8 | Composes the two largest gains; the quality budget is spent twice on the same slices and must be checked after each step. |
caveat The scores rank techniques for one device class and hide that "quality" is a per-slice quantity: two options with the same score can lose it on entirely different inputs. They also cannot express that the latency gain of quantization flips from small to large when the target is a memory-bound GPU rather than a compute-bound CPU.
Pick by the bound, spend by the slice
The plan for the phone came out in the opposite order from the naive attempt: distil first, because a large teacher and an unlabelled corpus existed and the compute bound needed fewer multiply-adds; then structured pruning of the student's least-used heads; then eight-bit weights for memory and cache. After each step, the slice report.
The quality budget was set per slice before the first step. The accent slices were given a tighter tolerance than the aggregate because the product team said so, and the sequence stopped when the next step would have spent more of that tolerance than it bought in latency.
What binds on the target device, and what do you have to work with?
when Bytes read per token set the speed; the model barely fits.
cost Weight quantization first — most of the gain for least effort; then a slice evaluation, because the loss hides on rare inputs.
when Multiply-adds set the speed; bytes are secondary.
cost A smaller architecture via distillation, or structured pruning; a training or fine-tuning run either way; quantization afterwards for memory.
when Whatever the bound, the small model can be taught rather than trained from scratch.
cost A teacher pass over the corpus and a student training run; the student inherits the teacher's errors and version.
when The model fits with headroom at the latency budget.
cost Do nothing. Compression spends quality for a gain nobody needed.
A compressed model is a new model
Everything the domain says about promotion applies: the compressed artifact needs its own evaluation on every slice, its own shadow or canary, its own entry in the registry with lineage to the teacher and the compression settings, and the original kept as a rollback rung. The temptation is to treat compression as a build step, and a build step does not get a slice report.
The assumption that outlives the compression is that the loss stays where it was measured. It does not, on its own: the teacher changes, the calibration sample is refreshed, the device kernel is updated, and each can move the loss to a slice nobody re-checked.
Each monitored slice of the compressed artifact stays within the tolerance set before compression, relative to the original, on data untouched by any compression step.
holds when The slice report was run on this artifact after the last step; the teacher, calibration sample and pruning criterion are recorded in lineage and unchanged; on-device kernels are the ones benchmarked.
breaks when A re-compression skips the report; the teacher is upgraded and the student is not re-distilled; a device or kernel update changes numerics; a new production slice appears that no report covers.
respond Roll back to the previous artifact for the affected population; re-run the compression sequence with the slice report after each step; do not widen the tolerance.
How to build it
Most important first.
- Start from the constraint and the device: measure what binds on the target hardware — bytes, multiply-adds, memory — and pick the technique that reduces that term. Quantization for a memory-bound target; structured pruning or a smaller architecture for a compute-bound one.
- Distil early when a large model and unlabelled data exist. A small student trained on the teacher's soft outputs over a large corpus is usually a better starting point than the same student trained on hard labels alone, and it is the only technique that gives the rare-intent slices more signal rather than less.
- Set the quality budget per slice before compressing, and spend it in order of gain per quality lost on the target device — measured, not assumed.
- Treat each candidate as a model promotion: full slice evaluation, shadow or on-device test, lineage recorded, and the previous artifact retained for rollback.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- On-device latency and memory for each candidate, on the target hardware, with the production input distribution — the numbers the constraint is stated in.
- Per-slice quality delta against the original, worst slice first. This decides whether a candidate is acceptable; the aggregate decides nothing.
- File size is the number that is easiest to report and least related to on-device speed — a pruned model with zeros compresses well and runs no faster.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The compressed artifact's per-slice quality stays within the budget set before compression, measured against the original on held-out data that no compression step touched.
- The target-device benchmark that justified the technique used production-shaped inputs, and the binding term — bytes or multiply-adds — has not changed with a device or kernel update.
- The lineage from compressed artifact back to teacher, pruning criterion and calibration sample is recorded, so a change to any of them re-triggers evaluation.
- Offline: for each candidate, the full slice report against the original plus on-device latency and memory at the production input distribution.
- Online: a staged on-device rollout with quality monitored by slice and the previous artifact as the rollback (Canary Rollout).
- Over time: re-run the slice report when the teacher, the calibration data or the device changes; compression artifacts drift from their teacher without anyone retraining them.
What can go wrong
- The quality budget is spent on the aggregate, and every technique takes a little from the same rare slices, so the combination collapses a slice that each step alone left within tolerance.
- The on-device benchmark uses short, clean utterances; production audio is longer and noisier, and the latency budget is blown on inputs the benchmark never contained.
- The distilled student is trained on teacher outputs from an old teacher version; the teacher is later improved and the student, still in production, silently encodes the old behaviour.
- Quantization is cheap and hardware-dependent in its speed gain; pruning is effective only when structured and needs fine-tuning to recover; distillation gives the best small model and costs a training run plus a teacher to run over a corpus.
- Combining techniques compounds quality loss on the same slices; spending the budget in sequence needs a slice evaluation after every step.
- Every compressed model is one more artifact to version, monitor and roll back, with a lineage to a teacher that will itself change.
- "Half the weights are zero, so it is twice as fast." It is twice as sparse. Dense kernels multiply zeros at full speed; only structured sparsity that the kernel skips is faster.
- "Quantize first, it is always the cheapest win." On a compute-bound device the bytes were not the problem; the win is small and the quality cost was still paid.
- "Train a small model from scratch; it is simpler than distillation." Simpler, and worse on rare cases, because the small model sees only the hard labels and the rare classes have few. The teacher's soft outputs are the extra signal a small model needs.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- SCALE-SPECIFICThe trade-offs assume a model that does not fit its target; for a model that fits with headroom, compression is engineering effort spent for nothing, and for a very large model the distillation cost — running the teacher over a corpus — can exceed the original training cost.
- MODEL-SPECIFICTransformers distil well and quantize with outlier care; convolutional networks prune structurally by channel with good hardware support; tree ensembles compress by depth and tree count and none of these three techniques applies to them directly.
- SIMPLIFIEDThe matrix scores are relative rankings for a typical mobile target and not measurements; on a server GPU the latency column for quantization and pruning changes rank, and any quality figures are for the shape of the argument.
Where the depth lives
This domain teaches the model and hands the rest off by name.