On-device AI fits private workflows

On-device AI is interesting because it changes where the model sits in the user’s life.

A hosted model is a service call. The app sends context away, waits, gets an answer back, and decides what to do next. That is fine for a lot of work. It is less fine when the work is reading local notes, classifying private photos, summarizing screenshots, or helping with files that were never meant to leave the device.

privacy is about movement

I do not think of private AI as a model property. I think of it as a data movement property.

What leaves the device? What gets logged? What gets retained? What gets synced later as a derived artifact? On-device inference reduces how often the raw material has to move. That is the whole appeal.

small models need narrow jobs

The local model is most useful when the job is narrow and the acceptance surface is clear.

• classify whether a note is a task, reference, or draft
• extract contact info from a card image
• suggest tags for local files
• redact obvious secrets before remote submission
• summarize a short transcript segment

I do not want a local model pretending to be a frontier model with a smaller footprint. I want it to be excellent at the small jobs it can actually own.

{
  "pipeline": [
    {"step": "capture", "model": "small local classifier"},
    {"step": "tag", "model": "small local extractor"},
    {"step": "escalate", "model": "remote general model", "only_if": "confidence_low"}
  ]
}

That is the shape I trust. Capture locally. Tag locally. Escalate only when the local pass cannot answer with enough confidence.

local state changes the contract

Private workflows also have to account for derived state: embeddings, summaries, tags, memories, logs, and cached outputs. Those can leak privacy even when the original file stays local.

That means the app needs to say what is in scope, what can sync, what can be deleted, and what expires at the end of the session. A local index that reads everything and remembers forever is not a privacy feature. It is just a quieter server round trip.

The hybrid pattern still makes the most sense to me: keep the private and fast steps local, then escalate when the user actually needs the bigger model or the broader context. On-device AI is useful when it keeps the model close to the work and the boundary close to the user.

That is not a niche benefit. It is the difference between a tool that helps with a private habit and a tool that quietly turns every habit into remote data.