Foundation Models & Fine-Tuning
Encoder and decoder families, pretrained models reused across tasks, transfer learning, fine-tuning and parameter-efficient adaptation — at the level of what changes and what it costs.
Encoder-only models turn text into a representation and are what you want for classification and embeddings; decoder-only models generate the next token; encoder–decoder models read one sequence and write another. Pick by what the output must be, not by which is newest.
A foundation model is pretrained once on broad data and reused across many tasks. "Pretrained" hides a training system you did not run, data you did not choose and an objective you did not pick — and what you inherit shows up as serving cost and as benchmark numbers that are not your task's numbers.
Pretrained model + task data → adapted model. The early layers carry structure that transfers; the late layers carry the old task. Freeze what transfers, train what does not, and expect the labelled-data requirement — and the split arithmetic — to change.
Fine-tuning changes the weights with your labelled data; prompting and retrieval change the inputs and leave the weights alone. The first is an ML Engineering job with training, evaluation and a new artifact; the second is Agentic Engineering. Knowing which one you need is most of the decision.
Keep the base frozen, train a small number of new parameters — an adapter, a low-rank update to a few weight matrices — and ship the delta. Many task adapters can share one base in memory, which changes what an artifact is and what serving looks like; the price is a quality ceiling the base sets.