Parameter-Efficient Fine-Tuning
Keep the base frozen, train a small number of new parameters — an adapter, a low-rank update to a few weight matrices — and ship the delta. Many task adapters can share one base in memory, which changes what an artifact is and what serving looks like; the price is a quality ceiling the base sets.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
How can a task be learned by training a tiny fraction of a model's parameters, what does that do to the artifact and to serving, and what quality does it give up?
A platform team serves one foundation model to twelve internal products, each of which now wants its own fine-tune. Twelve full copies of the weights would need twelve times the GPU memory the team has, and the platform lead has asked whether the products can each have "their own model" without each having their own GPU.
Fine-tune the base once per product. Twelve artifacts, twelve deployments, each product owns its model. Buy the GPUs.
Twelve full-size artifacts is twelve times the weight memory, which the fleet does not have; loading them on demand means a cold start of tens of seconds per switch, which no product's latency budget absorbs (Startup Time & Cold Start).
- Twelve full-size artifacts is twelve times the weight memory, which the fleet does not have; loading them on demand means a cold start of tens of seconds per switch, which no product's latency budget absorbs (Startup Time & Cold Start).
- Twelve full fine-tunes on small datasets each carry the forgetting risk of the last lesson, twelve times, with twelve evaluation sets to prove otherwise (Fine-Tuning).
- The base model gets a new version; twelve full fine-tunes must be re-run and re-evaluated, and the products with a few thousand examples did not need to touch most of the weights in the first place.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Each product predicts something different — a classification, an extraction, a formatted rewrite — with its own labelled set and its own evaluation; the platform's target is serving all twelve within one memory and latency budget.
- The decision for each product is whether a shared base plus a small adapter meets its quality bar, or whether it needs a full fine-tune and its own serving footprint.
- Twelve labelled sets from a few thousand to a few hundred thousand examples, of very different shapes; a single base model of tens of billions of parameters.
- The GPU fleet holds a handful of copies of the base; the memory bandwidth per token is the serving ceiling (Memory Bandwidth & VRAM).
- Products ship on different cadences; a product that retrains weekly must not force the other eleven through a promotion cycle.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A fine-tune changes a weight matrix W to W + ΔW. The observation behind low-rank adaptation is that ΔW for a task is close to low rank: it can be written as B·A with A of shape r×d and B of shape d×r, r a small number like 8 or 16, so the trainable parameters go from d² to 2·r·d — for d = 4,096 and r = 8, from about 17 million per matrix to about 65 thousand. Train A and B, keep W frozen; at inference the model computes W·x + B·(A·x), or merges B·A into W once.
- Adapters do the same thing with a different placement: small bottleneck layers inserted between the frozen blocks, trained while everything else is fixed. Either way the base never changes, the task-specific state is a small delta, and the gradient computation touches a tiny fraction of the parameters — so the training run needs far less memory for optimiser state and finishes far faster (Optimisers: SGD, Momentum, Adam).
- The serving consequence is the reason it matters here. One copy of the base in memory can serve many adapters: the per-token cost of the base is paid once and each request adds its own small B·A. Switching product is switching a few megabytes of adapter, not reloading tens of gigabytes. The artifact for a product is now the delta plus a hard pointer to the exact base version it was trained against (What a Model Artifact Contains).
A low-rank delta on a frozen base
Full fine-tuning updates W in place. The adapter view keeps W and learns the update separately as the product of two thin matrices, so the trainable state is a sliver of the original and the base is untouched. At inference the two paths add: the base's W·x and the adapter's B·A·x. Nothing about the base's computation changes, which is what makes sharing it possible.
The rank is the dial. Higher rank expresses more change and costs more parameters; the empirical finding that motivated the technique is that most task adaptations sit at a low rank, so a small r reaches close to full fine-tuning quality. When it does not, the plateau is the signal that the task needs more than a delta.
1import numpy as np2d, r = 4096, 83W = load_base_weight("attn.q_proj", base_version="v3.1") # d x d, FROZEN4A = np.random.normal(scale=0.01, size=(r, d)) # trainable: r x d5B = np.zeros((d, r)) # trainable: d x r, starts at 06 7def forward(x): # x: d8 return W @ x + B @ (A @ x) # base path + low-rank delta9 10# trainable parameters per matrix:11# full fine-tune : d*d = 16,777,21612# low-rank, r=8 : 2*r*d = 65,536 (~0.4%)13# B starts at zero so the adapter is the identity on step 0 — the base is14# exactly preserved until training moves it, which is the forgetting guard15 16# the artifact for this product is (A, B, base_version, rank, which matrices)Two things carry the lesson. B is initialised to zero, so at step zero the model is exactly the base — the adapter cannot forget what it has not yet changed. And the artifact line: A and B are useless without the base version they were trained against, which is why the version is part of the delta's identity.
What the artifact becomes
A full fine-tune produces a model: tens of gigabytes that stand alone. An adapter produces a delta: a few megabytes that mean nothing alone. That inverts the registry's assumptions. The unit of promotion is now (base version, adapter), the base is a shared dependency with its own lifecycle, and a base upgrade is a change to every adapter's artifact even though no adapter file changed (What a Model Artifact Contains, The Model Registry).
It also changes rollout. Rolling back a product is swapping a small file, in seconds, with no effect on the other eleven; rolling back the base is a fleet event. Canary per adapter, pin the base, and treat the base's promotion as the platform's most careful one (Canary Rollout, Rollback & Fallback).
The base weights resident in the serving process are byte-identical to the base each loaded adapter was trained against.
holds when The adapter artifact records the base's content hash, the server verifies it at load time, and a base upgrade is blocked until every adapter has been re-evaluated against the new base (Artifact Integrity).
breaks when The base is upgraded by naming convention ("latest"); a hot-fix to the base is applied in place; an adapter trained on a quantised base is served on the full-precision one or vice versa (Quantization).
respond Refuse to load, fall back to the previous base or to the prompted base for that product, and re-train the adapters against the new base before promoting it.
What it gives up
The adapter cannot change what the base fundamentally computes. If a product's task needs the model to read its input differently at a low level — a new format, a domain far from pretraining — a rank-8 delta on a few attention matrices may plateau below the bar, and no amount of retraining at that rank fixes it. The honest response is a measured comparison against a full fine-tune on a subset, and a promotion to a full artifact for that product if the gap is real.
The other price is coupling. One base serving twelve products is one dependency with twelve consumers: its upgrade, its outage and its capacity are shared. The platform gains memory and loses independence, and whether that is the right trade depends on how different the twelve products' bars and cadences are.
| Option | Quality | Latency | Cost | Interpretability | Data needed | Operational | Note |
|---|---|---|---|---|---|---|---|
| Prompted shared base, no training | Long prompts cost tokens on every request; behaviour the base resists stays wrong. | ||||||
| Adapter per product on a pinned base | Small artifacts, shared memory, per-product rollback; a base upgrade is twelve retrains. | ||||||
| Full fine-tune per product | Highest ceiling; twelve full-size artifacts the fleet cannot hold resident. | ||||||
| Small task-specific model per product | Right for the products whose task is narrow and latency-bound; no shared base at all. |
caveat The quality column assumes the low-rank regularity holds for each product's task, which is exactly the thing that has to be measured per product; and the cost column counts GPU memory, not the engineering of a multi-adapter serving layer, which is a real project.
How to build it
Most important first.
- Default to an adapter per product on a shared, pinned base; promote a product to a full fine-tune only when its evaluation shows the adapter ceiling is below its bar (Promotion Is a Checklist, Not a Score).
- Make the base version part of the adapter's identity: the registry entry records base version, rank, which matrices were adapted, the data version and the per-slice evaluation; an adapter is not promotable to a base it was not trained on (The Model Registry, Model Lineage).
- Serve with a multi-adapter server that keeps the base resident and hot-swaps adapters per request, and load-test the adapter switch rate, not just the per-request latency (Model Serving Architecture, Inference Batching).
- Evaluate each adapter on its own held-out set against two baselines: the prompted base and a full fine-tune on a subset, so the quality given up is a measured number rather than a suspicion.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Per-product quality on its held-out set against the full-fine-tune baseline; the gap is the price of the adapter and the number that decides promotion.
- Base weight memory per GPU and the count of adapters resident, plus p99 latency including an adapter switch — the numbers that say the fleet can serve twelve products.
- Do not measure by trainable-parameter count or training time alone; a fast, cheap adapter that misses the product's bar has saved nothing.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every adapter served is loaded against the exact base version it was trained on — checked at load time, not assumed from a naming convention.
- The task-specific change each product needs stays expressible as a low-rank update at the trained rank; a task that drifts toward needing a different reading of the input breaks this and shows as a plateau under retraining.
- The shared base's capacity is partitioned so that one product's traffic burst cannot consume the latency budget of another; the serving layer, not the model, enforces this (Bulkheads: Buying Independence by Giving Up Utilisation).
- Offline: for each adapter, report held-out quality next to the prompted base and a full fine-tune on the same split; assert the adapter's base hash matches the served base's hash in the promotion pipeline (Artifact Integrity).
- Online: canary each adapter behind its predecessor with per-product metrics and the shared p99, and roll back the adapter — a small file — rather than the base (Canary Rollout).
- Over time: on any base upgrade, re-evaluate every adapter against the new base before serving any of them; treat a base change as a change to all twelve artifacts.
What can go wrong
- The base is upgraded for one product's benefit; the other eleven adapters are now deltas against weights that no longer exist, and they degrade silently because the server still loads them (The Model Supply Chain).
- A product's task needs a genuinely different reading of the input — a new modality-like shift — and rank 8 cannot express it; the adapter plateaus below the bar and the team raises the rank repeatedly instead of promoting to a full fine-tune.
- The multi-adapter server batches requests across products; a burst from one product starves another of the shared base's capacity, and the latency SLO fails for a product whose traffic did not change (Throughput vs Latency).
- An adapter gives a small artifact, cheap training and shared serving at the price of a quality ceiling set by the base and the rank; where a task needs the base to change, it cannot.
- One base for twelve products is one blast radius: a base upgrade, a base outage or a base capacity problem is twelve incidents.
- Merging the adapter into the weights removes the per-request adapter overhead and gives back a full-size artifact per product — the memory problem returns.
- "An adapter is a smaller model." It is a small delta on a large model. It needs the base, in the exact version, to mean anything; the artifact is the pair, and the serving footprint is the base's.
- "Low rank means low quality." For most task adaptations the useful change is genuinely low rank and the adapter matches full fine-tuning within the evaluation's interval. The exceptions are real and show up as a plateau; measure, do not assume in either direction.
- "We can upgrade the base and keep the adapters." An adapter is a delta against specific weights. Against different weights it is noise added at the same positions. Every base change is a retrain of every adapter.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- SIMPLIFIEDParameter counts use one d×d matrix and a single rank for the shape of the argument; real configurations adapt several matrices per block with a scaling factor, dropout and sometimes quantised base weights, which change the numbers and the memory but not the delta-on-a-frozen-base structure.
- SCALE-SPECIFICThe serving argument — one base, many adapters — matters at a fleet serving several products from a large base; a team with one product and a small model can full-fine-tune and serve a single artifact with none of this machinery, and should.
- CONTESTEDA serious position holds that parameter-efficient methods are a compromise imposed by memory limits and that with enough compute a full fine-tune is simply better — the low-rank assumption is an empirical regularity, not a law, and it fails on tasks that need the model to learn new low-level structure. That is right for those tasks, and the lesson says to measure the gap; what the position underweights is that the quality difference is often within the evaluation interval while the artifact and serving difference is an order of magnitude, which for a shared platform is the decisive term.
Where the depth lives
This domain teaches the model and hands the rest off by name.