AI incident reviews need model context

AI incident reviews need model context because “the model failed” is not a root cause.

It is a starting complaint. Maybe a useful one. But if the review stops there, nobody learns anything operational. The team leaves with a mood instead of a fix.

AI incidents are annoying because the failure often crosses layers. The user sees a bad answer, a wrong action, a refusal that should not have happened, or an approval flow that did not show enough context. Underneath that, the cause may involve the prompt, model route, retrieved documents, tool schema, cache state, policy layer, evaluator, or a human approval that was too vague to be meaningful.

If the incident review only captures the final output, it is already missing the system.

the model version is not trivia

The first thing I want in an AI incident review is the exact model path.

Not the provider name. Not “the latest model.” The actual model or deployment identifier, the route that selected it, the fallback behavior, and whether anything changed recently. If a request can move between a cheap model, a reasoning model, a local classifier, and a judge model, the review needs to know which one made which decision.

This matters because “model behavior” is rarely one thing. A user-facing answer may come from one model. A safety decision may come from another. A tool-call argument may be generated by a third. A verifier may accept or reject the result. If the system has routing, the incident has routing context.

The review should ask:

• which model generated the user-visible output?
• which model selected the tool?
• which model judged the result?
• did routing use the intended path?
• did a fallback fire?
• did a model upgrade land near the incident?

Without that record, the team is stuck comparing memories.

prompts are versioned behavior

Prompts are code, or close enough to code that pretending otherwise gets expensive.

An incident review should include the prompt version that produced the behavior. If the prompt was assembled dynamically, the review should include the rendered prompt or at least the meaningful pieces: system instruction, developer instruction, user request, retrieved context, tool descriptions, and any policy text included in the call.

The rendered prompt matters because the bug may not live in the stable template. It may live in the assembly.

Maybe retrieval inserted stale context above newer context. Maybe the tool description changed. Maybe a safety instruction got placed after a long chunk and lost practical influence. Maybe the user request was summarized before the model saw it. Maybe the prompt included two competing instructions and the model picked the wrong one.

If the review cannot show what the model actually saw, it should say so. That is not a footnote. That is a finding.

retrieval has to be part of the timeline

For RAG systems, “the answer was wrong” is often a retrieval incident first.

The model may have answered reasonably from bad context. It may have missed the right document. It may have used an old chunk because the index was stale. It may have retrieved a document the user was not allowed to see. It may have received five chunks where the correct answer was in the sixth.

The incident review should include:

• query text or embedding input
• index version
• retrieved document IDs
• chunk timestamps
• ranking scores if available
• permission filters applied
• documents excluded by policy
• whether the correct source existed at the time

That last question is important. Sometimes the system could not have answered correctly because the source was missing. Sometimes the source existed and retrieval failed. Those are different fixes.

tool calls need receipts

Tool use adds another layer of failure. A model can call the right tool with the wrong argument. A tool can return a valid response that is operationally useless. The application can accept the response without checking whether it matched the user’s intent.

The review should capture the tool schema, arguments, returned value, and any validation or verification that happened afterward.

{
  "tool": "update_ticket_priority",
  "arguments": {
    "ticketId": "INC-1842",
    "priority": "high"
  },
  "result": {
    "status": "ok"
  },
  "verification": "ticket priority changed, no customer impact check"
}

The tool result saying “ok” is not the same as the workflow being ok. Incident reviews need to preserve that distinction.

approval state belongs in the review

Human approval is often treated as the end of the story. The user approved it, so the system did what it was told.

That is too convenient.

The review should ask what the user saw before approving. Did they see the target? The diff? The affected account? The rollback path? The risk? Did the approval apply to one action or a chain of actions? Did the agent do anything after approval that should have required a second approval?

An approval button without context is not a meaningful control. If an incident depends on a human approval, the review needs the approval prompt as it appeared at the time.

replay is the useful test

The check I care about most is replay.

Can the team replay the incident with the original model version, prompt, retrieved context, tool schema, route, and approval state? If not, which piece is missing?

Replay does not have to mean perfect determinism. Models are stochastic. Providers change infrastructure. Some systems cannot reproduce every token. Fine. The review can say that. But the team should still be able to reconstruct the decision path well enough to test the likely fixes.

If replay is impossible because logs were missing, prompts were not stored, retrieval snapshots were not preserved, or model versions were vague, that is part of the incident. The system failed to leave enough evidence.

the output is only the symptom

I would structure an AI incident review around the path from user request to user-visible failure:

1. What did the user ask for?
2. What context did the system select?
3. Which model or models handled the request?
4. Which tools were available and which were used?
5. What did the system verify?
6. What did the human approve?
7. What did the user see?
8. Which step first became wrong?

The last question is the one that matters. The visible failure may happen at the end, but the first wrong step often happens much earlier.

That is why model context belongs in the review. Not because every incident is the model’s fault. Because without the model context, you cannot tell whether it was.