Routing is the quiet part of AI platforms

The quietest part of an AI platform is often the part that decides where the request goes.

People like to talk about the model. That is understandable. The model is the visible miracle. It writes the answer, calls the tool, classifies the alert, summarizes the document, or gives you the weird failure that ruins your afternoon.

But production systems are full of decisions that happen before the model gets a turn.

Which context should be retrieved? Which model should receive the request? Should the system use a cheap classifier first? Should the request go to a local model because the data is private? Should the agent get tool access, or should it answer from context only? Should the product refuse because the user does not have permission to ask about that resource?

That is routing.

It is less dramatic than generation, but it shapes almost everything the user experiences.

attention is already a routing story

The name of this post started as a note to myself about attention.

Inside a transformer, attention decides which tokens matter to which other tokens. That is not the same thing as platform routing, obviously. I do not want to stretch the analogy until it snaps. But the instinct is useful: intelligence is not only in the final computation. A lot of the behavior comes from deciding what gets considered.

An AI product has its own version of that problem.

The request cannot look at every document, every tool, every model, every policy, and every memory with equal seriousness. The system has to narrow the field. It has to decide what belongs in the working set.

That happens in layers:

• route the user to the right product capability
• route the request to the right context source
• route the context to the right model shape
• route risky work through review
• route low-confidence output toward fallback

None of those choices are glamorous. They decide whether the glamorous part has a chance.

one path is a prototype smell

A prototype often has one path because one path is how you get something working.

Take the user’s prompt. Retrieve a few chunks. Send everything to the strongest model. Stream the answer. Log the transcript. Done.

That is a good way to learn. It is a bad permanent architecture.

Real traffic is uneven. Some requests are cheap. Some are sensitive. Some need fresh data. Some need a deterministic lookup. Some are mostly classification. Some need reasoning time. Some should never reach a frontier model because the data should stay local. Some should not run at all because the user is asking across a permission boundary.

If every request goes through the same path, the system spends too much money on easy work, too little care on risky work, and too much trust on ambiguous work.

The platform needs more than one path because users ask more than one kind of question.

context routing is usually the first failure

When an AI answer is wrong, people often blame the model first. Sometimes that is fair. Often the model was given the wrong room to search.

Context routing is the decision about which knowledge source, memory, file set, or retrieval index should be used for a request. It sounds simple until the product has more than one source of truth.

A support assistant might need public docs, private account data, internal runbooks, recent incident notes, and product changelogs. A coding agent might need source files, test output, package metadata, docs, and previous plan notes. A security workflow might need alert history, identity context, asset inventory, threat intel, and policy definitions.

Throwing all of that into one retrieval bucket is lazy. It also makes permission bugs easier to hide.

Context routing should ask boring questions:

• what resource did the user actually name?
• which sources are allowed for this user?
• which source is freshest for this kind of fact?
• does the request need private state or public docs?
• should old memory be ignored for this task?
• would missing context be safer than guessed context?

The last question matters. Sometimes the right route is to say the system cannot answer because the necessary context is unavailable. Retrieval is not a moral obligation. Bad context can be worse than no context.

model routing should be late enough to know the job

Choosing a model before understanding the task is usually premature.

The system should first classify the shape of work. Is this extraction, generation, reasoning, search, transformation, moderation, ranking, or tool planning? Does it need a long context window? Does it need low latency? Does it need deterministic structure? Does it involve sensitive data? Is the answer allowed to be approximate?

Only then does model choice become meaningful.

A small model may be better for a narrow classifier because it is fast, cheap, and easier to evaluate. A larger model may be worth the cost for messy reasoning or synthesis. A local model may be the only acceptable route for private notes. A specialized embedding model may matter more than the generator if the task is retrieval quality.

The wrong routing habit is “use the best model unless cost complains.”

The better habit is “use the model whose failure mode matches the task.”

That phrase is clunky, but it is how I actually think about it. A cheap model that fails loudly may be safer than a strong model that produces elegant nonsense. A slow model may be fine for a nightly review and unacceptable for autocomplete. A model with great code ability may be the wrong choice for a policy-sensitive support answer if the governance path is weak.

tool routing is a permission decision

Tool access should not be bundled into the model route by accident.

The platform might choose the same model for two requests but expose different tools based on user, task, environment, or risk. A local coding task may allow file reads. A reviewed edit may allow file writes. A deployment task may require a separate approval path. A user asking a general question should not inherit tool access because the previous request needed it.

This is where agent systems get messy. Tool routing combines capability and permission. The model may be capable of using a tool well, but the user may lack authority for the side effect. Or the user may have authority, but the current request may not warrant that tool.

I like routing that makes tool availability explicit:

request: summarize failing test output
context: current terminal log, related source files
model: coding-capable low-latency model
tools: read-only filesystem
blocked: shell execution, network, external issue tracker

That tells the system what kind of work this is. It also gives a reviewer a way to spot accidental privilege.

good routing leaves traces

Routing decisions should be visible after the fact.

If an answer fails, I want to know which path the request took. Which context source was chosen? Which model answered? Which tools were available? Which policy denied or allowed a route? Was there a fallback? Did the request hit a cost ceiling? Did the router think the task was summarization when it was actually diagnosis?

Without that trace, routing becomes folklore. People argue about model quality when the real bug was stale retrieval. They tune prompts when the request should have gone to a structured extractor. They blame latency on the model when the context builder fetched too much.

The trace does not need to be beautiful. It needs to be specific enough to replay the decision.

quiet systems still need owners

Routing work is easy to ignore because it rarely has a single screenshot.

There is no big button labeled “route better.” There is a collection of small choices that make the product feel smarter, cheaper, faster, safer, or more frustrating. That is why routing needs an owner. Otherwise every team adds a special case, every prompt starts carrying hidden policy, and every failure becomes somebody else’s layer.

The platform does not need to be elaborate on day one. Start with explicit routes. Log the decisions. Measure where requests go. Keep a few ugly examples where the route was wrong. Add learned routing only after the rules have proved too crude.

The useful AI systems I trust tend to have this habit: they know the difference between a hard problem and the wrong path for an easy one.

Routing is how they learn that difference.