Smaller models belong in the architecture

The largest model in a system should not be the default answer to every question.

That is not mainly a cost argument. Cost matters, but the more interesting part is architectural. Smaller models can sit closer to the work. They can run with lower latency, tighter permissions, smaller prompts, and sometimes without sending the data anywhere at all. They are also easier to scope. A small model that only classifies or extracts can be much easier to reason about than a general model that tries to help everywhere.

The hard part is not deciding that small models are useful. The hard part is deciding where they are allowed to own a step.

the job has to be narrow enough to name

Before I pick a model, I want the task written down in the dullest possible form.

input: one customer message, under 4,000 characters
output: { intent: billing | login | cancellation | bug | unknown }
abstention: allowed
latency budget: 50 ms p95 after tokenization
failure cost: one extra handoff
escalation: unknown routes to the larger model or a human

That kind of contract does more for model choice than a benchmark chart. It tells you whether the job is narrow enough for a smaller model to own it. It also makes the failure mode visible before the model ships.

If the task description is “understand the user,” the job is too vague. If the task is “extract invoice numbers and return null when absent,” the boundary is clear. The smaller the model, the more precise the boundary has to be.

distillation is mostly a data problem

Distillation gets talked about like it is a clever compression trick. In practice it is a data pipeline.

A larger teacher model produces labels, rankings, rationales, or rewrites. A student model learns from that output. If the task is narrow and the teacher is good enough, the student can become fast and cheap enough to use in hot paths.

The naive version is too simple:

ask big model for labels
train small model on labels
ship small model

That pipeline copies mistakes, hides uncertainty, and gives the student no reason to know where the edge of the task is. It can look good on a friendly eval set because the teacher shaped both the data and the style of the examples.

The version I trust is messier:

raw examples
  -> teacher labels
  -> confidence checks
  -> disagreement review on thin slices
  -> train, validation, and adversarial holdout splits
  -> student training
  -> per-slice eval against a baseline
  -> shadow rollout

The teacher helps create coverage. It does not get to grade itself.

the student should learn one behavior, not the whole workflow

A student model is not a compressed copy of the teacher’s entire skill set.

If the teacher can read a long support thread, infer sentiment, choose a policy path, and write a reply, the student should probably do one of those jobs. Maybe it classifies the intent. Maybe it extracts the order number. Maybe it decides whether the request needs policy retrieval. If you ask the student to inherit the entire workflow, you usually get a weaker assistant with a stronger opinion about its own competence.

I would rather split the work across small, boring models:

• a router that picks billing, account, bug, or unknown
• an extractor that finds IDs and timestamps
• a redactor that removes secrets before logs are stored
• a similarity model that finds nearby historical cases
• a guard model that blocks a narrow set of risky requests

Each of those can have its own eval and fallback. That makes the architecture easier to change without rebuilding the whole product.

baselines keep the project honest

The first comparison should be embarrassingly simple.

Regexes, keyword rules, embedding nearest-neighbor search, and prompted large models are all useful baselines. If a distilled student cannot beat a cheap rule on the slices that matter, it has not earned a place in the system.

For routing, I would compare:

• a keyword baseline
• an embedding baseline
• a prompted large model
• the trained student
• the student with abstention enabled

The abstention case matters because a small model is often valuable when it knows when to step aside. A model that gets 92 percent accuracy by guessing every example may be worse than one that gets 86 percent coverage with high precision on the cases it accepts.

I also want slice-level reporting. Rare classes, ambiguous inputs, long inputs, multilingual examples, and adversarial examples all deserve their own line. A small model can look great overall and still fail the exact slice that justified the work.

latency changes the product surface

Small models can make different interfaces possible.

If an intent classifier runs in tens of milliseconds, the UI can update suggestions as the user types. If a redaction model runs locally, the app can protect logs before they leave the device. If a router is cheap, the system can retry or ask for a second opinion without turning every request into a budget discussion.

That is not a minor implementation detail. It changes what feels natural to the user.

But low latency also tempts teams to put the model everywhere. Then the small model shows up in hot paths, background jobs, moderation, and analytics. If its behavior changes, the whole system changes with it.

So the small model needs the same release discipline as anything else in the stack: versioning, evals, fallback paths, and a clear note about what changed.

where small models fail

Small models fail at the boundaries.

They overfit label language. They learn formatting clues from synthetic data. They collapse rare classes into common ones. They become stale when user behavior changes. They can also be harder to debug than rules because the failure is distributed across weights, thresholds, and training data.

The worst failure is clean confidence. A large model may ramble when it is unsure. A small classifier may return a neat enum with no visible doubt. That looks tidy and it can be very wrong.

The escape hatches I want are simple:

• an unknown class
• calibrated thresholds
• a second model for disagreement on high-risk cases
• random audit sampling
• shadow comparison against a larger route

None of that makes the student less useful. It just keeps the system honest.

the architecture i actually want

I do not want a system that asks whether the frontier model can be replaced.

I want a system that asks:

• which steps are narrow enough to model directly?
• which steps need general reasoning?
• which steps need privacy more than raw capability?
• which steps can abstain safely?
• which steps are called often enough that latency dominates?
• which steps have labels good enough to train against?

That is where smaller models belong. Use the large model where ambiguity and context justify the cost. Use the small model where the job is repeated, bounded, and measurable. Use distillation when the teacher can help create a better dataset, but keep the eval independent enough to catch copied mistakes.

The point is not efficiency as a slogan. The point is to put the right amount of intelligence in the right part of the system.