Quantization
Storing weights in fewer bits — FP32 to FP16, BF16, INT8 — shrinks memory and speeds up memory-bound inference. The quality cost is real, concentrated on rare inputs, and only visible if you evaluate on the same slices you used before.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The quantized model is half the size and passes the aggregate evaluation. Where would a quality loss hide, and how do I know it did not?
A document-classification model is being quantized from thirty-two-bit floats to eight-bit integers to fit two more copies per device. The aggregate validation metric moved by a hair. A week after rollout, the legal team reports that contracts in one under-represented language are being misfiled — and they were fine before.
Quantize with the framework's one-liner, check that the overall metric barely moved, ship. Rounding weights to eight bits is a small perturbation and neural networks are robust to small perturbations.
The perturbation is small on average and large where the weight distribution has outliers — and the outliers encode the rare cases. A per-tensor scale wide enough to represent the largest weight rounds the small ones to nothing.
- The perturbation is small on average and large where the weight distribution has outliers — and the outliers encode the rare cases. A per-tensor scale wide enough to represent the largest weight rounds the small ones to nothing.
- The aggregate metric is dominated by the common language. A collapse on the rare one moves it by a hair, which is exactly what was observed and dismissed.
- Calibration on validation data makes the validation metric optimistic for the quantized model specifically; the slice that was not in the calibration sample got no such favour.
- Nothing failed. The model is smaller, faster, and confidently wrong on the documents the legal team cares about most.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The surrounding model assigns documents to categories; the quantization's target is the same decisions at a fraction of the memory and time, with the quality loss bounded and known per slice.
- The decision to ship the quantized model is a comparison: quality on every slice that matters against the previous model, not one aggregate number against itself.
- The model was trained in thirty-two-bit floating point. The quantized version stores weights and, in the aggressive variant, activations as eight-bit integers with a per-tensor or per-channel scale.
- Calibration data — a few hundred documents used to choose the scales — was sampled from the validation set by whoever ran the conversion, which touched the set that was supposed to be held out.
- The evaluation slices from the original model launch exist, by language and document type, but the quantization was signed off on the overall metric alone.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Quantization maps a range of real values onto a small integer grid:
q = round(x / scale) + zero_point, withscalechosen so the tensor's range fits the grid. FP32 to FP16 keeps a floating-point format with a smaller exponent and mantissa; BF16 keeps FP32's exponent range with fewer mantissa bits, which is why it trains more stably than FP16; INT8 abandons the exponent entirely and relies on the scale. - The rounding error per weight is bounded by half a step of the grid. What matters is the error relative to the weight's magnitude: with one scale per tensor, weights far smaller than the tensor's maximum are rounded to a few steps or to zero. Per-channel scales, and separating outlier channels, reduce this at some complexity cost.
- Post-training quantization converts a trained model and uses calibration data to choose scales; it is cheap and works well when the weight and activation distributions are well-behaved. Quantization-aware training simulates the rounding during training so the weights adapt to it; it costs a training run and recovers most of the loss where post-training does not.
- The speed gain comes from two places: fewer bytes read per token, which is the whole gain on a memory-bound pass (Memory Bandwidth & VRAM), and integer arithmetic units that many devices provide at higher throughput than floating-point ones. The first is nearly guaranteed; the second depends on the hardware and the kernel.
Rounding is not uniform in its effects
The quantized weight is round(x / scale), and the scale is set so the tensor's largest magnitude fits the grid. Every weight then carries an absolute error of up to half a step. For weights near the maximum that is a small relative error; for weights a hundred times smaller it is a large one, or total — they round to zero.
The small weights are not noise. In a classifier they are often the connections that distinguish a rare class from a common one, because the rare class contributed few gradient updates. Quantization erases them first, which is why the loss concentrates exactly where the aggregate metric cannot see it.
1import numpy as np2 3def quantize_int8(w):4 scale = np.abs(w).max() / 127.0 # one scale for the whole tensor5 q = np.clip(np.round(w / scale), -127, 127).astype(np.int8)6 return q, scale7 8def dequantize(q, scale):9 return q.astype(np.float32) * scale10 11w = np.concatenate([np.random.normal(0, 0.01, 990), np.array([2.0] * 10)]) # outliers12q, s = quantize_int8(w)13err = np.abs(dequantize(q, s) - w) / (np.abs(w) + 1e-9)14# relative error on the 990 small weights is enormous: the step is 2.0/12715# ~ 0.016, larger than most of them, so most round to zero.16# per-channel scales, or keeping the 10 outliers in FP16, fixes it.The ten outliers set the scale for the other nine hundred and ninety. This is the mechanism behind per-channel quantization and outlier-splitting schemes: keep the scale local so the small weights keep their resolution.
Evaluate on the same slices, or not at all
The original model was launched with a slice report: quality by language, by document type, by length bucket. The quantized model is a different model and needs the same report, compared side by side. An aggregate that moved by a hair is consistent with a rare slice that collapsed, and the report is the only way to tell the two apart.
The offline/online gap here is the ordinary one for this domain, with an unusual cause: the artifact changed without the training data or the features changing, so nothing in the monitoring stack was looking for a step change on rollout.
Aggregate validation metric within noise of the original model; conversion signed off on that number alone; calibration sample drawn from validation.
Misfiling reports from the legal team for contracts in one under-represented language; no alert fired because no monitor was sliced by language.
- 1Per-tensor scales set by outlier weights erased the small weights that distinguished the rare language's category boundaries, so quality collapsed on that slice while the dominant language was untouched.
- 2The aggregate metric is weighted by slice size and cannot show a collapse on a small slice; the per-slice report that would have shown it was not run.
- 3Calibration on validation data made the quantized model's validation number slightly optimistic, and the rare slice was under-represented in the calibration sample so its activation ranges were mis-scaled.
Post-training versus quantization-aware
Post-training quantization takes a finished model, picks scales from a calibration sample, and rounds. It costs minutes and no training data. Quantization-aware training inserts simulated rounding into the forward pass during training or fine-tuning, so the weights learn to sit where rounding hurts least; it costs a training run and recovers most of what post-training loses on hard slices.
The choice is by the slice report: if post-training quantization is within tolerance on every slice, stop; if not, try local scales and outlier handling; if still not, quantization-aware training or a higher precision for the affected tensors.
For every monitored slice, the quantized artifact's quality is within tolerance of the original's, on held-out data untouched by calibration.
holds when Scales are local enough that small weights keep resolution; calibration came from training data and covered the rare slices; the slice evaluation was run for this artifact.
breaks when A re-quantization uses a new calibration sample and skips the slice report; a fine-tune changes the weight distribution under the old scales; a new slice appears in production that no report covers.
respond Roll back to the higher-precision artifact for the affected slice or entirely; fix scales or retrain quantization-aware; do not raise the tolerance to make the report pass.
How to build it
Most important first.
- Evaluate the quantized model on the same slices as the original — by language, document type, length, and every subgroup the launch review cared about — and set a per-slice tolerance, not an aggregate one (Evaluation Slices).
- Draw calibration data from the training distribution, never from the validation or test set, so the held-out evaluation stays held out. Stratify calibration across the rare slices, since the scales are chosen from what calibration shows.
- Choose the precision by the bound: if the pass is memory-bound, weight-only quantization captures most of the gain at the least risk; activation quantization adds risk for a compute-bound gain that may not exist on the device.
- Where a slice degrades past tolerance, try per-channel scales and outlier handling before quantization-aware training; keep the higher-precision model as the fallback rung (Serving Fallbacks).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Per-slice quality delta between quantized and original, with the worst slice reported first. This is the number that decides shipping; the aggregate delta is the number that hides the failure.
- Bytes per parameter, tokens or documents per second, and device memory after conversion — the numbers the quantization was for, to check the gain is real on this hardware.
- Agreement rate between quantized and original predictions on a held-out sample is cheap and useful as a smoke test, and insufficient as a sign-off because disagreements cluster.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Quality on every monitored slice stays within the tolerance the launch review set, checked against the original model on the same held-out data, for this quantized artifact specifically.
- The calibration sample was drawn from training data and covered the rare slices, so the scales represent the inputs production will send and the validation set remains untouched.
- The speed and memory gains were measured on the production device and kernel, and a device or kernel change re-triggers the measurement.
- Offline: run the full slice evaluation on both models; fail promotion if any slice exceeds tolerance, whatever the aggregate says.
- Online: shadow the quantized model against the original for a period and log disagreements by slice (Shadow Deployment); the clusters are where quality moved.
- Over time: re-run the slice evaluation after every re-quantization, fine-tune, or device change, and monitor the rare slices' prediction distribution for a step change at rollout.
What can go wrong
- Per-slice evaluation is done at launch and not repeated after the next quantization, which used a different calibration sample and different scales; the process was validated, the artifact was not.
- The INT8 kernels on the production device are slower than the FP16 ones for this shape, so the model is smaller and no faster — the memory gain was real, the compute gain was assumed.
- A quantized model is fine-tuned later in full precision and re-quantized with the old scales, which no longer fit the new weight distribution.
- Post-training quantization is cheap and loses more on hard slices; quantization-aware training recovers quality at the cost of a training run and a more complex pipeline.
- Per-channel scales and outlier handling protect rare weights at some kernel complexity and, on some hardware, some of the speed gain.
- A thorough slice evaluation is a full evaluation pass per quantization, which is the cost that gets skipped when the aggregate metric looks fine.
- "The overall metric barely moved, so quantization is lossless here." It moved a little in aggregate because a small slice moved a lot. Lossless is a per-slice claim.
- "INT8 is four times faster than FP32." It reads a quarter of the bytes, which is the gain on a memory-bound pass; the arithmetic speed-up depends on the device having fast integer units and the kernel using them.
- "We calibrated on validation data because it is representative." It is; that is why it was held out. Calibration is a fitting step, and fitting on the validation set makes the validation number optimistic.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- MODEL-SPECIFICLarge transformers tolerate eight-bit and lower weight quantization well in aggregate but have outlier channels that need per-channel or mixed handling; small convolutional networks quantize cleanly; models with narrow decision margins — calibrated risk scores near a threshold — flip decisions on rounding that would be invisible in a ranking task.
- SIMPLIFIEDThe quantization formula shown is symmetric per-tensor affine quantization; real schemes use per-channel, group-wise and mixed-precision variants whose details the hardware domain covers. Any quality figures in this lesson are for the shape of the argument, not measured.
- CONTESTEDSome practitioners hold that weight-only post-training quantization to eight bits is effectively lossless for large models and that the slice evaluation is ceremony for a change that never bites; their evidence is strong on aggregate benchmarks. The counter is that "never bites" has been observed on the slices people measured, and that the rare-input concentration of the error is exactly what an aggregate benchmark cannot see — so the evaluation is cheap insurance against a loss that is by construction invisible without it.
Where the depth lives
This domain teaches the model and hands the rest off by name.