Evals are how AI teams stay honest

Evals are where AI work stops being vibes.

That sounds harsher than I mean it, but only slightly. A team can spend a surprising amount of time arguing from screenshots. One prompt feels better. One model sounds cleaner. One retrieval tweak looks promising in the five examples everyone keeps reusing. Then the change ships and a user finds the failure case in ten minutes.

The answer is not to stop using judgment. Judgment is still required. The answer is to give judgment something repeatable to push against.

An eval is a feedback loop. It takes examples the team cares about, runs the system against them, grades the result, and makes regressions visible before users become the test harness.

pick the unit first

The first eval question is not “which metric?” It is “what is the unit?”

For a chat assistant, the unit might be one turn, a whole conversation, or a task completed across several tool calls. For a classifier, it might be one input and one label. For retrieval, it might be one query and a set of expected documents. For an agent, it might be an entire run with intermediate steps, final output, and side effects.

Choosing the wrong unit makes the eval lie.

If the product promise is “the agent updates the issue and leaves a useful comment,” grading only the final text is too narrow. The agent might write a good summary while editing the wrong file. If the promise is “find the right source document,” grading only the generated answer hides retrieval failure. The model may guess correctly once and fail when the wording changes.

I like writing the unit down in a shape close to the system:

type EvalCase = {
  id: string
  input: string
  context?: string[]
  expectedBehavior: string
  forbiddenBehavior?: string[]
  tags: string[]
  grader: "exact" | "schema" | "retrieval" | "llm" | "human"
}

That is intentionally plain. The value is not in the type definition. The value is forcing the team to say what the case is testing before the model runs.

baselines beat impressions

Every eval needs a baseline. Without one, a score is a floating number.

The baseline might be the current production prompt, the current model, a rule-based system, a human-written answer set, or a previous checkpoint. The baseline gives the team a reference point for a change. Did the candidate improve the target behavior? Did it regress something boring but important? Did it buy quality by spending too much latency or cost?

The baseline also keeps the team from grading against memory. People remember the dramatic failures and the impressive wins. They forget the ordinary cases. A baseline run is less emotional.

For a prompt change, I want to see:

baseline: support-answer-prod-2024-08-01
candidate: support-answer-citations-2024-08-13
change: require citations for policy claims
expected improvement: fewer unsupported refund answers
watch: more refusals on ordinary billing questions

Now the eval has a theory. The candidate is not “better” in the abstract. It is supposed to improve one behavior without damaging another.

rubrics should be boring and specific

LLM output is messy enough already. The rubric should not add poetry.

Bad rubric:

Grade whether the answer is high quality, helpful, accurate, and clear.

Better rubric:

Pass only if:
- the answer uses the cancellation policy from the provided context
- the answer does not promise a refund
- the answer tells the user which page to open
- the answer asks for account verification before discussing billing details

The better rubric is more annoying to write. That is why it works. It names the product behavior.

Some evals can be deterministic. JSON schema validity, required fields, exact labels, forbidden strings, citation presence, and tool argument shape should not need an LLM judge. Use deterministic checks when the requirement is deterministic.

LLM judges are useful when the requirement is semantic: faithfulness, tone, completeness, or whether a response actually answered the question. But the judge still needs a tight rubric, sampled calibration, and human review for disputed cases. Otherwise the team starts optimizing for the judge’s taste.

slices matter more than averages

The average score is usually where the meeting starts. It should not be where the decision ends.

AI systems regress in slices. Long inputs. Short inputs. Old documents. Non-English text. Mixed intent. Rare labels. Tool failures. Users who ask indirectly. Inputs with logs. Inputs with sarcasm. Inputs copied from another system.

If those slices are not labeled, the eval cannot protect them.

I like tagging cases with the reason they exist:

{
  "id": "billing-ambiguous-017",
  "tags": ["billing", "ambiguous-intent", "policy", "high-risk"],
  "input": "I was charged again after I thought I cancelled. Can you fix it?",
  "expectedBehavior": "Explain that account verification is required before billing details. Do not promise a refund."
}

That high-risk tag is doing work. If the model improves the overall suite while regressing high-risk billing cases, the release should slow down. The aggregate score can be green and the decision can still be no.

examples should come from production pain

Synthetic cases are useful, but the best eval suites usually start with real failures.

A user asked something the system mishandled. A reviewer rejected a generated answer. A tool call used the wrong argument. A retrieval result pointed at stale docs. An agent made a plausible change and missed the required cleanup. Those examples are gold because they already proved the system can fail there.

When a bug happens, the fix should include an eval case if the behavior might come back. This is normal software testing discipline, applied to AI systems. A regression test should survive the incident.

The workflow can be small:

1. Capture the failed input and relevant context.
2. Remove private data.
3. Write the expected behavior.
4. Add tags that explain why the case matters.
5. Run the baseline and candidate.
6. Keep the case in the suite unless it becomes obsolete.

This is how the eval suite becomes institutional memory. It remembers the weird things users actually did instead of only the examples the team imagined.

evals should enter the release path

An eval that runs only when someone remembers is a notebook, not a guardrail.

The right amount of automation depends on the risk. A small prompt tweak for an internal helper might run a fast smoke suite in CI and a larger suite before deploy. A customer-facing support assistant might require passing critical slices before rollout. An agent that can mutate files or trigger workflows may need replay tests, sandbox runs, and human approval for new capability.

The gate should be explicit:

block release if:
- high-risk slice drops by more than 1 percent
- any deterministic safety case fails
- p95 latency increases by more than 20 percent
- cost per successful task increases by more than 30 percent
warn if:
- low-risk slice drops by more than 3 percent
- judge disagreement rises above 10 percent

Those numbers are examples, not universal truth. The useful part is the shape. The eval decides what blocks, what warns, and what needs review.

That prevents a common failure: everyone sees a regression, but nobody knows whether it is severe enough to stop the release.

evals need maintenance

An eval suite can rot.

Docs change. Policies change. User behavior changes. Models learn around old examples. A case that used to be hard becomes trivial. A case that used to be valid now encodes an outdated product rule. The suite can also become too large, too slow, or too expensive to run often.

Maintenance is part of the work. Delete obsolete cases. Add new failures. Split slow suites from smoke suites. Track flaky grader results. Review slices that never fail because they may no longer be testing anything meaningful.

I would rather have a smaller eval suite that people trust than a giant suite everyone ignores.

the feedback loop changes behavior

The point of evals is not the score. The point is how the team behaves because the score exists.

Good evals make engineers write narrower prompts because vague prompts fail specific cases. They make product decisions more concrete because rubrics need expected behavior. They make model upgrades less superstitious because candidates are compared against baselines. They make incidents more useful because failures become regression tests. They make demos less misleading because there is a pile of ugly examples waiting behind the happy path.

That is why evals keep AI teams honest. They do not remove judgment. They make judgment answer to evidence.