Smaller models belong in the architecture

The largest model should not own every step.

Smaller models can run closer to the work, with lower latency, narrower permissions, shorter prompts, and sometimes no external data transfer. The hard design question is where a small model is allowed to make a decision.

Start with a contract that is boring enough to test:

input: one customer message, under 4,000 characters
output: intent enum or unknown
abstention: allowed
latency budget: 50 ms p95
failure cost: one extra handoff
escalation: larger model or human

If the task is “understand the user,” the boundary is too vague. If it is “extract an invoice number and return null when absent,” the job is measurable.

Distillation is mostly a data pipeline. A teacher can create labels, rankings, or rewrites, but a naive pipeline copies mistakes and hides uncertainty:

raw examples
  -> teacher labels
  -> confidence checks
  -> disagreement review on thin slices
  -> train and holdout splits
  -> student training
  -> per-slice eval
  -> shadow rollout

The teacher helps create coverage. It does not get to grade itself.

The student should learn one behavior, not the teacher’s whole workflow. A support system may use separate models for intent routing, ID extraction, redaction, similarity search, and narrow risk checks. Each can have its own evaluation and fallback.

Simple baselines keep the project honest. Compare the student with rules, embeddings, a prompted large model, and an abstaining variant. A model that guesses every example may score higher than one that accepts fewer cases with much better precision. That does not make it more useful.

Report the slices that matter: rare classes, ambiguous inputs, long inputs, multilingual examples, adversarial cases, and accepted versus escalated requests.

Latency can change the product surface. A local redaction model can protect logs before they leave a device. A fast router can update suggestions while the user types. Those benefits also make it tempting to put the model everywhere, so versioning, thresholds, fallback paths, and release evals still matter.

Small models fail at boundaries. They overfit synthetic label language, collapse rare classes, become stale, and return clean confidence when the answer should be unknown. Keep the escape hatches visible:

• unknown class
• calibrated threshold
• disagreement route for high-risk cases
• random audit sampling
• shadow comparison with a larger model

The goal is not to replace the frontier model. It is to put general reasoning where ambiguity justifies it and smaller, measurable behavior where repetition and latency dominate. Distillation earns its place when it produces a better bounded component, not merely a cheaper imitation.