Queues make AI workflows calmer

Queues make AI workflows calmer because they stop pretending every request should finish inside the user’s patience window.

As soon as a task needs retrieval, model calls, tool execution, retries, file processing, verifier passes, or human review, a synchronous request starts lying about the shape of the work. Some work should stay inline. A short classification, a route decision, or a small rewrite can still fit in the request path. The rest needs somewhere honest to live.

The first thing I want is identity.

{
  "jobId": "summarize-ticket-1842",
  "idempotencyKey": "ticket-1842:v3",
  "attempt": 2,
  "visibleAfter": "2026-07-13T12:10:00Z",
  "deadLetterAfter": 5
}

The idempotency key matters because AI jobs sit behind flaky systems all the time. The model call may succeed while the worker crashes before recording the result. A tool may finish while the network drops. The user may hit the button again. The queue may redeliver after a visibility timeout. Identity is what keeps that from turning into duplicate work.

I also like input references more than giant payloads in the queue message. Put the big document, image, prompt bundle, or repository snapshot somewhere durable. The job points to it. That makes retries read the same input version instead of whatever happened to be in memory when the worker woke up.

retries need reasons

“Retry on failure” is too crude.

Some failures are transient, like a provider timeout or a network issue. Some are permanent until the input changes, like malformed schema, missing permission, deleted source material, or a verifier that rejected the result for a deterministic reason. Those should not share the same loop.

type JobFailure =
  | { kind: "transient"; retryAfterMs: number; reason: string }
  | { kind: "input"; reason: string }
  | { kind: "permission"; reason: string }
  | { kind: "policy"; reason: string }
  | { kind: "verifier"; reason: string }
  | { kind: "bug"; reason: string }

dead letters are a review queue

Dead-letter queues are not failure closets. They are evidence that the workflow does not know what to do with a case. Better to inspect them and learn than to let them disappear into a counter nobody reads.

backpressure is the point

A queue also gives the system a place to say “not yet.” That is healthier than letting every agent workflow compete for model capacity, database connections, reviewer attention, and budget at once.

The calm part is not that queues make AI easy. The calm part is that they give slow, expensive, failure-prone work a shape software already knows how to operate.