SWE-bench stopped being enough

SWE-bench was useful because it made coding models answer to code.

That was a big step up from vibes. A model could talk a good game in chat and still fail to patch a real issue. SWE-bench forced the question into a repository with tests and a target behavior.

The problem is that every useful benchmark eventually becomes less useful as a separator.

saturation is a success state

A benchmark getting saturated is not a failure of the benchmark.

It means the field got better at the thing it measured. Models improved. Tooling adapted. Agents learned to patch more realistic issues. The leaderboard got crowded because the task became part of the evaluation culture.

That is progress.

The mistake is treating the benchmark as a permanent oracle.

Once several systems score well, the small differences may not predict local usefulness. A model that wins by a few points may still be worse in your repo because it edits too broadly, ignores conventions, or handles flaky tests badly.

local repos have local weirdness

Every serious codebase has texture.

There are naming conventions, generated files, service boundaries, lint rules, migration habits, and commands nobody runs casually. A coding agent has to navigate that texture.

Public benchmarks rarely know:

• which tests are flaky
• which generated files should not be edited
• which directories are owned by another team
• which migrations need rollback
• which helper APIs should be used
• which style choices reviewers care about
• which commands are too expensive to run casually

Those details decide whether the agent feels helpful or reckless.

local evals should use real tasks

The best local coding evals come from real work.

Closed bugs. Small refactors. Dependency updates. Test failures. UI fixes. Documentation edits. Migration reviews. Security patches. The eval should contain tasks your team actually wants agents to handle.

For each task, capture:

• starting commit
• task prompt
• allowed tools
• expected behavior
• required checks
• known pitfalls
• reviewer rubric

The goal is not to recreate SWE-bench badly. The goal is to measure local usefulness.

tests pass is necessary and insufficient

Tests matter. If the agent cannot get the test green, that is a clear signal.

But “tests pass” is not the whole review. A coding agent can still touch too many files, introduce unnecessary abstraction, ignore existing helpers, or hide a behavior change inside a cleanup.

I would score:

• correctness
• patch size
• convention fit
• test relevance
• explanation quality
• command choice
• failure recovery
• reviewer cleanup required

That last one is underrated. If the agent saves twenty minutes of typing and creates thirty minutes of cleanup, the benchmark pass is not the product result.

traces belong in the eval

For coding agents, evaluate the run as well as the final diff.

Did the agent read the right files? Did it inspect failing tests? Did it run a targeted check before a broad one? Did it notice a command failure? Did it keep trying the same broken path? Did it ask for missing context? Did it stop when the task was done?

The trace shows whether the system is operable.

the local eval is the product eval

For coding agents, the product is not “can produce code.” The product is “can move this repo safely.”

That means local evals should include the boring parts: setup, commands, flaky tests, repo conventions, reviewer expectations, and failure recovery.

SWE-bench made coding models more serious. The next step is making evaluation closer to the work.

When the public benchmark stops being enough, that is not a reason to trust vibes again. It is a reason to build the eval harness the repo already needed.