Reasoning models changed where latency belongs

The o1 preview made a useful distinction easier to see: a workflow does not need to spend the same amount of time on every step.

For a long time, latency discussions collapsed into one number. Lower p95 was good. Streaming earlier was good. A response that arrived now was preferable to one that arrived later. Those are sensible goals for autocomplete and simple transformations. They become less useful when the task includes uncertainty, tool calls, or a costly mistake.

Different work earns different waits

A person does not experience latency as a dashboard metric. They experience it as a promise from the interface.

Two hundred milliseconds is already enough to make autocomplete feel wrong. Five seconds for a careful code review may be reasonable. A background analysis can take longer still, provided the product tells the user that work is happening and gives them a way to return to it.

The right question is not “How fast is the model?” It is “What kind of wait does this job justify?”

Route by difficulty

The expensive model is not automatically the correct model. Some requests are hard. Some are merely long. Some are small transformations. Some need external evidence. Others need a cautious refusal more than an elaborate chain of reasoning.

Routing should follow the shape of the work. A simple rewrite should not wait behind a deep analysis, and a high-stakes review should not be forced through the same budget as a quick lookup.

routes:
  quick_rewrite:
    max_latency_ms: 1500
    reasoning: none

  code_failure_analysis:
    max_latency_ms: 20000
    reasoning: high

  legal_policy_question:
    max_latency_ms: 30000
    reasoning: high
    fallback: human_review

These limits are product decisions as much as model settings. They describe how much uncertainty the system is allowed to carry before it returns an answer, asks for help, or stops.

Progress needs a better signal

Streaming makes a slow response feel active, but tokens are not necessarily progress. A user may care more about seeing that the system is reading the relevant context, checking a source, running a tool, or reviewing its result.

That suggests a different interface for slower work: expose workflow milestones where they are useful, and keep the model’s internal delay from becoming the only status indicator.

The practical rule is simple: spend latency where uncertainty is expensive. Waiting is easier to accept when the extra time has a job.