Multimodal systems need grounded workflows

An image in a chat box is useful. It is not yet a workflow.

A workflow has source material, extracted artifacts, a review point, and a clear response when the model and evidence disagree.

Consider a bug report with a screenshot, console log, user note, and screen recording. Those inputs have different failure modes:

type Evidence =
  | { kind: "screenshot"; assetId: string; source: "user_upload" | "system_capture" }
  | { kind: "console_log"; assetId: string; source: "browser" | "server" }
  | { kind: "transcript"; assetId: string; derivedFrom: string }
  | { kind: "user_claim"; text: string; source: "form_field" }

Keep them separate so the answer can point to the evidence it used. Combining everything into one prompt makes review guesswork.

Extraction is a boundary, not the source of truth. OCR can merge columns. Speech-to-text can miss acronyms. Frame sampling can skip the moment the UI changed. Chart extraction can catch the title and miss the axis scale.

Contradictions should remain visible:

{
  "status": "needs_review",
  "reason": "user claim conflicts with screenshot"
}

A user may say a button is disabled while the screenshot shows it enabled. OCR may report one total while the image shows another. The right result may be review, not a confident synthesis.

Grounding is also an interface problem. If the model says an invoice is wrong, show the relevant region and extracted values. If it cites a video, link to the timestamp. If it uses a chart, show the series and scale.

A multimodal system earns trust when it can say what it saw, what it extracted, and when the evidence was too thin to decide.