Fine-tuning is not a shortcut

Fine-tuning is usually an attempt to make a decision permanent inside the model.

That is useful only after the decision is clear. Freeze the base model, prompt, retrieval path, eval set, scoring rule, and known failures first. The question is not whether the checkpoint is “better.” It is what behavior should change.

Maybe the model misses security escalations, normalizes messy entities, emits invalid JSON, or needs a huge prompt because the workflow was never simplified. If the target cannot fit in one sentence, training will encode the ambiguity rather than resolve it.

The dataset should teach one job. A narrow classifier needs hard negatives and explicit labels. A formatter needs valid and invalid examples. A style model needs real examples of the intended style, not generic professionalism. Synthetic data can help, but it belongs outside the holdout and should not become the model’s only voice.

A useful training record names the boundary:

task: classify support escalations
base model: frontier-small-2026-05
prompt version: escalation-v4
train set: reviewed production tickets + hard negatives
holdout: later tickets + hand-written edge cases
primary metric: macro f1
guardrails: no regression on billing, vip, or ambiguous policy

Then compare slices rather than one headline score:

slice                  baseline   ckpt_a   ckpt_b
security_high_recall      0.82      0.88     0.91
billing_precision         0.90      0.87     0.79
ambiguous_policy_f1       0.64      0.63     0.56
json_validity             0.93      0.97     0.98
out_of_domain_f1          0.71      0.69     0.60

Checkpoint B wins one row and loses the product.

The same applies to structured actions. If the model creates a ticket, the expected shape should be explicit:

{
  "task": "create_ticket",
  "args": {
    "title": "Parser fails on quoted commas",
    "priority": "high",
    "owner": "platform"
  }
}

Fine-tunes often break at adjacent slices. A support router learns the label vocabulary without learning the boundary between security and billing. An output becomes polished while losing caution. A model becomes too eager to sound certain.

Keep neighboring checks, calibration, lineage, and rollback. The payoff of a successful fine-tune is simplification: less prompt, less repair, less latency, or less ceremony. If the training data is still arguing about the task, a checkpoint will only store the argument.