Fine-tuning is not a shortcut

Fine-tuning is the part people reach for when the decision already exists and they want the model to absorb it.

decide the task first

If the team is still arguing about what the task is, training will not settle that argument. It will just encode the argument badly.

That is why I want the boring parts frozen first: the base model, the prompt, the retrieval path if there is one, the eval set, the scoring rule, and the failures we already know about.

The useful question is not whether the checkpoint is “better.” It is what should change.

Maybe the model should stop missing security escalations. Maybe it should normalize messy entities. Maybe it should produce valid JSON without a repair loop. Maybe it should replace a huge prompt that exists only because the workflow never got cleaned up.

The moment you can write down the target behavior in one sentence, you can start checking whether training is the right lever at all. If the sentence is still mushy, the model will not rescue you from that. It will only make the mushy part harder to unwind later.

I also want a baseline that stays boring on purpose. A frozen prompt, a fixed retrieval path, and a named evaluation slice tell me what fine-tuning is trying to beat. Without that, the checkpoint comparison is just theater.

let the data teach one thing

That choice determines the data.

A narrow classifier wants ugly negatives and explicit label definitions. A formatter wants valid and invalid examples. A style model wants real examples of the style, not generic professionalism. Synthetic data can help, but only if it stays out of the holdout and does not teach the model the generator’s accent.

For a support router, I want the training set to look like the thing the model will actually see:

security_high: 420
security_medium: 610
billing_escalation: 980
product_bug: 740
unknown: 300

That table says more than a paragraph of optimism. It tells me where the model will get lazy. It tells me whether the rare class is rare enough to need a weighting trick or a better label definition.

For a task like support escalation routing, I want the training rows to expose the edge, not just the happy path:

task: classify support escalations
base model: frontier-small-2026-05
prompt version: escalation-v4
train set: reviewed production tickets + hard negatives
holdout: later tickets + hand-written edge cases
primary metric: macro f1
guardrails: no regression on billing, vip, or ambiguous policy

Then I want a small eval table that can actually surprise me:

slice                  baseline   ckpt_a   ckpt_b
security_high_recall      0.82      0.88     0.91
billing_precision         0.90      0.87     0.79
ambiguous_policy_f1       0.64      0.63     0.56
json_validity             0.93      0.97     0.98
out_of_domain_f1          0.71      0.69     0.60

Checkpoint B looks nice if you only care about the headline slice. It is a bad pick if billing noise or ambiguous policy matters to the product.

One concrete example helps here. If the model is supposed to create a ticket, I want the record to carry the exact output shape and not just a vague natural-language instruction:

{
  "task": "create_ticket",
  "args": {
    "title": "Parser fails on quoted commas",
    "priority": "high",
    "owner": "platform"
  }
}

That kind of example teaches the structure and the behavior together. If the model can only guess at the shape, the team still needs repair code. If the model can output the shape but not the class, the label set is wrong.

where it usually breaks

This is where fine-tunes usually go wrong. The target slice improves, then the adjacent slice gets weird. Or the output looks polished while losing caution. Or the model starts mirroring the label language instead of the task itself.

That is why I keep adjacent checks around. Not because they are elegant. Because they catch the model when it has learned the wrong lesson.

One concrete failure mode is the model that learns the naming pattern but not the decision boundary. A support router can look excellent on “security” versus “billing” and still fail on the tickets that mention both. The metric goes up while the product gets harder to trust.

If the fine-tune works, the payoff is simplification: less prompt, less repair, less latency, less ceremony.

If the team is training because labels are fuzzy or the behavior is disputed, the checkpoint will not fix that. It will just store the confusion in weights.

The thing I watch next is calibration. A checkpoint can be right and still be annoying if it becomes too eager to sound certain in the exact places where the workflow wants a second look. That usually means the model learned the naming pattern but not the product boundary.

Once it ships, treat it like a versioned dependency. Record the lineage. Keep the rollback. Note why this checkpoint won. Otherwise the next person is left guessing why the model suddenly got a little different.