AI apps need boring observability

“Bad answer” is a real report and an incomplete trace.

An AI application needs ordinary logs, request IDs, and timings plus prompt versions, model versions, retrieval results, tool calls, parsed output, and the result the product accepted.

A run record can start small:

{
  "runId": "run_01J9",
  "feature": "support_summary",
  "model": "provider/model-version",
  "promptVersion": "support-summary-system@8",
  "retrievalQueryId": "qry_8831",
  "toolCount": 2
}

Retrieval needs its own trace because retrieval failures look like model failures. The wrong chunk may return. The right chunk may be filtered by permissions. A reranker may bury the useful source. Without the retrieval trail, the team blames the model for a context problem.

Tool calls need the same treatment: tool name, arguments, permission check, duration, result, and failure. Otherwise an agent bug becomes an argument about whether the tool, model, or application hid the evidence.

Metrics should stay sliceable. Inspect long inputs, model versions, routes, user tiers, and policy-sensitive cases rather than averaging them into one number. Production examples worth reviewing should become fixtures and eventually eval cases.

The design goal is intentionally dull. When an AI system cannot answer “what actually happened?” it is not observable enough to trust.