SWE-bench stopped being enough

SWE-bench did something important: it made coding models answer to code.

A model could sound competent in a chat window and still fail to change a real repository. SWE-bench put the model in a project with tests, dependencies, and a target behavior. That was a useful correction.

The next correction is accepting that a public benchmark cannot stay a perfect proxy for every repository.

Saturation is a signal

When a benchmark becomes crowded, that does not mean it failed. It means the field learned to optimize for the thing being measured. Models improved, agents got better at repository-level patches, and the task became part of the evaluation culture.

The mistake is treating the leaderboard as a permanent oracle.

Once several systems perform well, a small score difference may tell you little about a particular codebase. The system with the higher score might still make broader edits, miss local conventions, or recover poorly from a flaky test.

Repositories have texture

Every serious codebase contains rules that a public task cannot fully capture:

• generated files that should never be edited directly
• service boundaries that are architectural boundaries too
• migrations with a required rollback path
• lint and test commands with very different costs
• helper APIs that reviewers expect agents to reuse
• flaky tests whose failures need interpretation
• directories owned by a different team

Those details are not incidental. They determine whether an agent can move the repository safely.

Make the local task real

The strongest local eval tasks are usually recovered from actual work:

• a closed bug
• a small refactor
• a dependency update
• a test failure
• a documentation change
• a UI fix
• a migration review
• a security patch

For each task, record the starting commit, prompt, allowed tools, expected behavior, required checks, known pitfalls, and reviewer rubric. The point is not to rebuild a public benchmark with fewer examples. The point is to measure whether an agent can handle the work your team actually has.

Green tests are one measurement

Tests are necessary. They are not sufficient.

A patch can pass while touching too many files, introducing an abstraction nobody asked for, bypassing an existing helper, or hiding a behavior change inside a cleanup. I would score the final result on correctness, patch size, convention fit, test relevance, explanation quality, command choice, and failure recovery.

Reviewer cleanup deserves its own line item. Saving twenty minutes of typing and creating thirty minutes of cleanup is a negative product result, even when the benchmark says pass.

The run itself also contains evidence. Did the agent read the relevant files? Did it inspect the failing test? Did it try a targeted check before an expensive one? Did it notice a command failure? Did it keep repeating a broken approach? Did it stop when the task was complete?

The trace shows whether the system is operable, not just whether the last diff compiles.

SWE-bench made coding systems more serious by attaching them to real code. When its scores stop separating the systems you are considering, the answer is not to return to vibes. Build an eval around the repository already in front of you.