Fine-Tuning
Fine-tuning changes the weights with your labelled data; prompting and retrieval change the inputs and leave the weights alone. The first is an ML Engineering job with training, evaluation and a new artifact; the second is Agentic Engineering. Knowing which one you need is most of the decision.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
When does a task need the weights changed, how is that done without destroying what the pretrained model knew, and where is the line between fine-tuning and adapting the input instead?
A legal-operations team uses a general language model to classify incoming contract clauses into forty categories the firm defines. With a carefully written prompt it gets most of them; a handful of categories that hinge on firm-specific conventions it gets wrong consistently. Someone wants to "fine-tune it"; someone else wants to "give it more examples in the prompt"; and the platform team wants to know which of those is their job.
The model is smart; if it gets a category wrong, explain the category better in the prompt and add examples. If that is not enough, fine-tune it — that is what fine-tuning is for.
The firm-specific categories are not a matter of explanation. They are a convention that contradicts the model's general reading of the clause, and a longer prompt buys a few points at the cost of tokens on every request — forever — while the mistakes on the convention-bound categories stay (Context Engineering).
- The firm-specific categories are not a matter of explanation. They are a convention that contradicts the model's general reading of the clause, and a longer prompt buys a few points at the cost of tokens on every request — forever — while the mistakes on the convention-bound categories stay (Context Engineering).
- Fine-tuning the whole model on forty thousand clauses with the default learning rate produces a classifier that is excellent on the fine-tuning set and has lost general language ability the rare categories relied on: it now misclassifies clause types it used to get right without help. The offline number on a random split of clauses looked better than the prompt ever did.
- The revised category guide means a fraction of the labels are wrong by today's definition. Fine-tuning learns them faithfully. The prompt-based approach never saw them.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict the firm's clause category from the clause text; the label is the category a paralegal assigned in the document management system; the decision is which review checklist the clause goes to.
- Forty classes, some rare, several distinguished only by firm convention — the same clause text is category A at this firm and category B at another.
- Around forty thousand historically labelled clauses, unevenly distributed across the forty categories, with a few hundred in the rarest (Class Imbalance).
- Labels drift at the edges: the firm revised its category guide twice in the last three years, so older labels for a few categories mean something different (Label Quality).
- The general model is a decoder-only foundation model available both as a hosted API and as open weights of a smaller size.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Fine-tuning continues training the pretrained weights on task data with a task loss — cross-entropy over the forty categories, or the next-token loss over a formatted label. Gradient descent moves every unfrozen weight toward the new objective. With a small learning rate and few epochs the weights move a little and keep most of what they knew; with a large rate or many epochs they move far and the pretrained knowledge is overwritten — catastrophic forgetting is the generic name for that (Transfer Learning).
- What fine-tuning changes is the function itself: the model's response to inputs it has never seen shifts, including inputs the fine-tuning set did not cover. That is its power — it can install a convention the general model would never infer from a prompt — and its risk, because the shift is global and only your evaluation set says where it went.
- Prompting and retrieval leave the weights alone and change what the model is shown: instructions, examples, retrieved passages. They change the output through the model's existing in-context behaviour, per request, at token cost per request, and they can be edited without a training run. They cannot make the model do what its weights cannot do; they can only steer what it can already do (RAG Overview, Context Construction & Grounding).
Weights or inputs
The decision is a question about where the fix lives. If the model would get the category right given the right information — the definition, three examples, the relevant clause from the guide — then the fix is information, and information is an input. Put it in the prompt, or retrieve it; that layer is editable without a training run and belongs to the agentic side of the boundary.
If the model reads the clause the general way and the firm reads it a different way, no amount of information changes the reading reliably; the behaviour has to move, and behaviour is weights. That is a fine-tune, and it comes with everything a model comes with: a split, a training run, an evaluation by slice, an artifact and a monitor. The platform team's question has an answer — the prompt is theirs, the weights are ours.
The categories the fine-tune targeted fail under prompting because the firm's convention contradicts the base model's reading, not because the prompt lacked a definition or example.
holds when A well-constructed prompt with the current guide and several examples per category was evaluated first and the convention-bound categories still failed consistently.
breaks when The prompt evaluation was skipped or weak; the categories fail because their definition is genuinely ambiguous, which no fine-tune resolves; the guide changes and what was a convention is now information.
respond If the prompt baseline matches the fine-tune, retire the fine-tune — it costs an artifact and an evaluation loop for nothing; if both fail, fix the category guide.
Adds a few hundred tokens per request; improves the convention-bound categories a little; the mistakes that contradict the model's general reading remain; editable in an afternoon.
No extra tokens per request; installs the convention in the weights; convention-bound categories improve substantially; the untouched categories are measured for forgetting; a registered artifact.
The failure was a convention contradicting the model's prior, which is a behaviour and lives in the weights. The prompt was steering behaviour the model does not have; the fine-tune changes the behaviour. Where the failure had been missing information, the ordering would reverse.
Changing the weights without losing what they knew
Fine-tuning is training, with the starting point being the pretrained weights instead of random ones. Everything from optimisation applies — learning rate, batch size, epochs, early stopping — with one twist: the starting point is valuable, and the goal is to move away from it as little as the task allows. A learning rate that is fine from random initialisation walks the model out of the pretrained basin in a few hundred steps.
The global nature of the shift is what makes evaluation different. A prompt change affects requests that contain the prompt. A weight change affects every input, including the clause types the fine-tuning set never contained. The held-out set therefore needs slices the fine-tune did not target, and the number on those slices is the one that says whether forgetting happened.
1# start from the pretrained weights, not from random2model = load_pretrained(base_version="v3.1") # pin it — the adapter is meaningless otherwise3 4# task loss over the firm's 40 categories5def loss(logits, y):6 return cross_entropy(logits, y, weight=class_weights) # rare categories up-weighted7 8# conservative: 1e-5 where pretraining used ~1e-4 .. 1e-3; 2-3 epochs; stop early9opt = AdamW(model.parameters(), lr=1e-5, weight_decay=0.01)10for epoch in range(3):11 for batch in train_loader: # split by DOCUMENT, not by clause12 opt.zero_grad(); loss(model(batch.x), batch.y).backward(); opt.step()13 if held_out(model, targeted).recall_stalled() or held_out(model, untouched).recall_dropped():14 break # the second condition is the forgetting checkTwo held-out checks in the stopping rule: one for the categories the fine-tune targets, one for the categories it must not damage. Most fine-tuning code has only the first, which is how forgetting ships with a green evaluation.
A fine-tune is a model; a prompt is a configuration
Once the weights move, the result is a model artifact with everything that implies: a base version it depends on, a data version it was trained on, an evaluation report by slice, a registry entry, a promotion decision and a monitor. Treating it as "the same model with a tweak" is how a deprecated base model takes the fine-tune with it, and how a guide revision silently invalidates the artifact (What a Model Artifact Contains, Promotion Is a Checklist, Not a Score).
The prompt, by contrast, is versioned configuration: it can be diffed, rolled back and edited by the team that owns the application. That asymmetry is the practical content of the boundary. Agentic Engineering versions prompts and retrieval pipelines; ML Engineering versions weights and adapters — and the two link to each other by base-model version (Prompts and Models Are Deployables on the DevOps side, Parameter-Efficient Fine-Tuning for the adapter form).
Aggregate accuracy on a random clause split rose clearly over the prompt-based baseline; the convention-bound categories improved most.
Paralegals reported new clause types being confidently filed under wrong categories, and two previously reliable categories degrading; the review checklists received clauses they had never received before.
- 1The learning rate moved the weights far from the pretrained solution and the model lost general reading ability on categories the fine-tuning set covered thinly.
- 2The random clause split placed near-duplicate clauses from the same contract on both sides, so the offline gain was partly memorisation.
- 3The fine-tuning set had no "other" class, so out-of-scope clause types could only land inside the forty.
How to build it
Most important first.
- Draw the boundary first. If the fix is information — a definition, an example, a retrieved document — it is an input change and belongs to the prompt or retrieval layer; Agentic Engineering owns that. If the fix is behaviour the model will not adopt from context — a convention, a format, a domain reading — it is a weight change, and this domain owns it with the full loop: split, train, evaluate, artifact, monitor (The Boundary With Agentic Engineering).
- Fine-tune conservatively: a learning rate one to two orders of magnitude below pretraining, few epochs, and a held-out set that includes the categories you did *not* fine-tune for, so forgetting is measured rather than assumed (Early Stopping).
- Clean the labels to the current guide before training, or restrict to post-revision examples; fine-tuning is the most faithful way to learn a stale definition (Label Construction).
- Produce a real artifact: weights or an adapter, the evaluation report by category, the data version, the base model version — registered and promotable like any model (The Model Registry, Model Lineage). A prompt is a configuration; a fine-tune is a model.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Per-category precision and recall on a held-out set split by document, with the convention-bound categories and the untouched categories reported separately — the second group is the forgetting measurement.
- Cost per classified clause under each approach, including the prompt tokens the long-prompt approach pays on every request (Inference Cost).
- Do not measure fine-tuning by aggregate accuracy on a random clause split. Clauses from the same contract are near-duplicates, and the aggregate hides the rare categories where the decision lives (Entity Leakage, Accuracy Under Imbalance).
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The firm's category conventions, which are what the fine-tune installed in the weights, remain the conventions; a guide revision changes the target and the fine-tune must be redone, not re-prompted.
- The base model version underneath the fine-tune is fixed; an adapter or delta trained against one base is meaningless against another (Parameter-Efficient Fine-Tuning).
- The clause text distribution at serving time matches the fine-tuning set closely enough that the global shift the fine-tune introduced does not misfire on clause types it never saw — checked on the untouched-category slice.
- Offline: evaluate prompt-only, prompt-plus-examples and fine-tuned on the same document-held-out set, per category; the fine-tune must win on the convention-bound categories and not lose on the rest.
- Online: shadow the fine-tuned model behind the prompt-based one, log disagreements, and have paralegals adjudicate a sample per category weekly (Shadow Deployment).
- Over time: keep a regression set that includes clause types outside the forty categories and re-score it on every fine-tune; monitor the "none of the above" rate in production as a forgetting signal (Model Regression Tests).
What can go wrong
- The fine-tuned model nails the forty categories and has lost the ability to say "none of the above"; a clause type that did not exist in training is confidently filed under the nearest category.
- The firm revises the guide again; the prompt-based classifier is updated in an afternoon and the fine-tuned one needs relabelling, a training run and a promotion cycle.
- The hosted API's fine-tuning endpoint returns a model id; the base model is later deprecated and the fine-tune with it, with no weights the firm holds (The Model Supply Chain).
- Fine-tuning removes the per-request prompt cost and installs behaviour prompting cannot, at the cost of a training loop, an evaluation set, an artifact to manage and a global shift you must measure.
- Prompting is cheap to change and expensive per request; it is the right tool while the definition is still moving and the wrong one once volume makes tokens the bill.
- A conservative learning rate preserves general ability and may leave the convention-bound categories short of what an aggressive fine-tune would reach; the untouched-category slice is the price tag.
- "Fine-tuning is how you teach the model new facts." It is how you change its behaviour. Facts that change — a new clause type, a new regulation — belong in the input via retrieval, where they can be updated without a training run. Fine-tuning them in makes them stale on the day the training set was cut.
- "Prompting is just a cheaper fine-tune." They act on different things — inputs against weights. A prompt cannot install a convention the model reads the other way; a fine-tune cannot be edited in an afternoon.
- "The fine-tuned model is better because its accuracy on our clauses went up." On a random clause split that number is partly memorised contract text, and it says nothing about the categories the fine-tune did not target — which is where it can have got worse.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat fine-tuning changes the weights and prompting changes the inputs — and that the first is a global shift measured only by your evaluation set — holds for every foundation model and every task; what is task-specific is which of the two the fix needs.
- CONTESTEDA serious position holds that for most teams fine-tuning is a mistake: prompt and retrieval engineering with a strong base model reaches the needed quality, keeps the system editable, avoids a training loop and an artifact to manage, and lets the base model improve underneath you. That is right where the fix is information or steering, and it fails where the behaviour contradicts the base model's reading, where per-request token cost dominates at volume, or where latency cannot afford a long prompt — and the only way to know is to evaluate both on the same held-out set.
- MODEL-SPECIFICLearning-rate ranges and forgetting behaviour differ between full fine-tuning of a decoder-only model and training a classification head on an encoder; the encoder route with a per-class head is often the better fit for a fixed forty-way classification and is the cheaper artifact to serve (Encoder / Decoder Families).
Where the depth lives
This domain teaches the model and hands the rest off by name.