Local models are part of the platform

Local models are part of the platform when they own a narrow job instead of pretending to be the whole assistant.

The useful jobs are boring in the best way: redact, classify, route, summarize, or help offline. That only works if the app treats the model as a real dependency with versioning, fallback, telemetry, and a privacy boundary.

what local actually buys

Small models are not simpler just because they are small. They move the work closer to the data, which means the platform has to stay honest about what gets stored, what gets logged, and what happens when the model is unsure.

That matters in practice. A local model can classify a note before it ever touches a remote service. It can summarize a document while the user is offline. It can pick a route for a request without sending private context out to a bigger model. The savings are not only latency. They are control.

a small router is still a router

I would still give the local path a real contract:

{
  "task": "classify_note",
  "input": "raw note text",
  "model": "phi-3-mini",
  "fallback": "remote-summary",
  "retain": false
}

That is enough to make the choice visible. The app can log the model version, record when it fell back, and decide whether the local result is final or just a first pass.

the boundary has to stay explicit

The moment the model touches private data, the product has to answer a few boring questions directly:

• what is stored locally
• what is allowed to sync later
• what gets discarded after the run
• what gets escalated to a larger model

If the answers are fuzzy, the system is pretending privacy is a feeling instead of a contract.

I like local models when they make the system more deliberate. The app can keep the quick stuff close, ship the hard stuff outward, and make the handoff obvious enough that a user can tell which path was taken.