The Signal Is at the End of the Document
Decide what you would do from the brief alone, including whether you would change anything at all. Everything below it is available, but the exercise stops working if you open it first.
A fine-tuned transformer classifies incoming legal documents into handling queues. It performs well on short documents and poorly on long ones (illustrative), and the long ones are the expensive ones to misroute. The proposal is to switch to a larger pretrained model.
Switching to a larger pretrained model of the same family and fine-tuning again. The aggregate accuracy improves a little — the larger model is better at short documents too — and the change is easy to justify because bigger models are supposed to be better. The context length did not change, the truncation did not change, and long-document accuracy does not move, at double the serving cost. The plot of accuracy against length, which would have taken ten minutes, would have shown that the two models fail at the same token count.
Read this even if you are confident. It is here rather than behind a button because it is the answer most teams actually ship, it passes review, and its cost arrives weeks later when the labels do.