Evals are how AI teams stay honest

Evals are the part of AI work that keeps a good demo from becoming a bad release.

A model can sound better while the system gets worse. A prompt can read cleaner while retrieval gets stale. The useful eval is not a scoreboard. It is a file cabinet for important behaviors, previous failures, and cases the team does not want to forget.

Start with the failure that matters. A support assistant may need to verify identity before discussing billing. A retrieval system may need the right source instead of a fluent guess. An agent may need to edit one file, leave a receipt, and stop.

The case should name that behavior:

type EvalCase = {
  id: string
  input: string
  context?: string[]
  expectedBehavior: string
  forbiddenBehavior?: string[]
  tags: string[]
  grader: "exact" | "schema" | "retrieval" | "llm" | "human"
}

Every candidate needs a reference point: production, a previous model, a rule-based system, or a human answer set.

baseline: support-answer-prod-2024-08-01
candidate: support-answer-citations-2024-08-13
change: require citations for policy claims
expected: fewer unsupported refund answers
watch: more refusals on ordinary billing questions

The rubric should be tighter than intuition. Deterministic requirements should use deterministic checks: schema validity, required fields, forbidden strings, citation presence, and tool argument shape. Use a model judge for semantic judgment, not because a model is available.

Tag the slices that fail: long inputs, old documents, non-English text, mixed intent, rare labels, tool failures, and high-risk policy cases. Aggregate scores can rise while one important slice gets worse.

The strongest cases usually come from real failures. Capture the input and context, remove private data, write the expected behavior, tag the risk, and keep the case until the behavior is no longer relevant. That is regression testing with a model-shaped surface.

Make the release gate explicit:

block if:
  high-risk slice drops more than 1 percent
  deterministic safety case fails
  p95 latency rises more than 20 percent

warn if:
  low-risk slice drops more than 3 percent
  grader disagreement rises above 10 percent

The thresholds will vary. The value is deciding in advance what blocks, what warns, and what needs review.

Evals also need maintenance. Policies change. Cases become obsolete. Graders drift. Suites become too slow to run. Delete what no longer protects the product, add what recently failed, and keep the suite small enough that people still trust it.

Evals do not remove judgment. They force judgment to answer to evidence instead of memory or the last impressive demo.