Eval dashboards for real feedback

An eval dashboard should make it harder to argue from vibes.

A top-line score is useful for scanning. It is not enough for deciding whether a change improved the product. A card showing 91.4% should lead quickly to the suite, case count, model and prompt versions, changed slices, and examples that flipped.

The underlying row needs enough context to debug:

type EvalCaseResult = {
  runId: string
  caseId: string
  suite: string
  slice: string[]
  inputRef: string
  expected: string
  output: string
  passed: boolean
  grader: "deterministic" | "llm" | "human"
  score?: number
  latencyMs: number
  costUsd?: number
  model: string
  promptVersion?: string
  retrieverVersion?: string
}

The dashboard summarizes this data. It should not replace it.

Aggregate scores hide regressions. A support assistant can improve overall while getting worse on refund policy. A code agent can pass more benchmark tasks and fail more dependency updates. Slices such as product area, language, risk, route, source, prompt family, input length, and model family make those changes visible.

A small table is often enough:

slice             cases   current   previous   delta
refund_policy       82     0.841      0.927   -0.086
login_recovery     144     0.965      0.951   +0.014
mixed_intent        39     0.692      0.718   -0.026

Failures need neighborhoods. A list of hundreds of failed cases is transparent and hard to use. Cluster likely causes such as missing retrieval, stale source, schema failure, bad tool arguments, timeout, refusal, rubric disagreement, or vague output. Deterministic validators can label some clusters; human labels are still useful for the rest.

A failure can become work:

case: refund-042
result: fail
reason: cited old cancellation policy
owner: docs
action: update retrieval freshness filter

Treat eval runs like release artifacts. Compare a candidate to a named production baseline and show pass-to-fail flips, fail-to-pass flips, changed output with the same grade, changed grade with the same output, and skipped or newly added cases.

Quality, latency, and cost belong in the same view. A one-point quality gain that doubles latency may be wrong for the product. A cheaper route that moves failures into high-risk cases is not a win. Useful distributions include p50, p95, and p99 latency, cost per accepted case, retry count, timeout rate, token use, and quality by latency bucket.

Grader disagreement is evidence, not noise. Show grader type and version, rubric version, human overrides, repeated-judge variance for sampled cases, and examples where graders disagree. If reviewers cannot agree on whether a case passed, the expected behavior may need clarification before the prompt needs another revision.

The dashboard is useful when it ends with a next action: open the examples, assign the failure cluster, inspect the route, revise the rubric, or promote the candidate with a watch list.

The point is not to declare that the model is better. It is to show better where, worse where, cheaper for which route, and more brittle under which inputs.