Reasoning models changed where latency belongs

The o1 preview made it easier to say something that should have been obvious for a while: some steps deserve more time than others.

Before reasoning models became a normal product category, latency conversation was too flat. Faster was better. Lower p95 was better. Stream sooner. Answer now. If quality was bad, try a bigger model or a better prompt. That is not wrong, just incomplete.

latency is part of the interface

A user does not experience latency as a metric. They experience it as a promise. A 200ms delay in autocomplete feels broken. A five-second delay for a careful code review can feel fine. A longer delay for a background analysis job can also be fine if the interface makes the wait legible.

route by difficulty, not prestige

The fastest way to misuse a reasoning model is to route everything through it because it is the “smart” model. Some tasks are hard. Some are just wordy. Some are small transformations. Some need external evidence. Some need careful refusal. The route should match the shape of the work.

hard steps need visible budgets

Reasoning time should have a budget. If the task is simple, keep it fast. If the task is risky, spend more time, but name the limit and the fallback. When the budget is gone, the system should ask for help, degrade, or stop.

routes:
  quick_rewrite:
    max_latency_ms: 1500
    reasoning: none

  code_failure_analysis:
    max_latency_ms: 20000
    reasoning: high

  legal_policy_question:
    max_latency_ms: 30000
    reasoning: high
    fallback: human_review

streaming is not the same as progress

Token streaming can make a slow model feel alive, but it does not always tell the user what is actually happening. Sometimes the useful progress is at the workflow level: reading context, searching sources, checking examples, running tools, or reviewing the result.

The rule I keep coming back to is simple. Spend latency where uncertainty is expensive. That is a better use of waiting than pretending every answer should arrive at the same speed.