Agent runbooks need failure examples

A runbook that says “restart the job” is not much help when the job completed and still did the wrong thing.

Agent failures often look like partial success. The file was edited, but it was the wrong file. The tool returned ok, but the argument pointed at yesterday’s record. The test passed, but it covered a different workspace.

I want runbooks to show the bad run first.

For example:

symptom:
  the agent says the deployment note is updated

evidence:
  the diff changed docs/deploy-old.md
  the active branch uses docs/deploy.md

wrong assumption:
  filename similarity was treated as ownership

recovery:
  stop the run
  restore the wrong file
  confirm the active path with a human
  rerun with the file scope explicit

That is much more useful than “context error.” It gives the reviewer something recognizable to look for.

I separate retries into two groups.

Mechanical retries are usually safe:

tool timeout
transient network error
rate limit with backoff
temporary file lock

Meaning retries are not:

wrong artifact selected
ambiguous request
valid but useless tool output
verification ran on the wrong surface
approval did not match the side effect

The second list needs a new question, a narrower scope, or a human decision. Repeating the same plan only makes the mistake more established.

A runbook should also say what evidence survives the run. For a coding agent, I want the diff, commands, checks, files inspected, approvals, and known gaps. The final line may be the most important one:

known risk: production deploy command was not tested

That is not an embarrassment to hide. It is information for the next person.

The best runbooks teach reviewers what to distrust: generated files, stale context, suspiciously broad diffs, tests that ran in the wrong package, and summaries that mention changes absent from the diff.

A short runbook with five real failure examples beats a long one made of abstract warnings.