Update 1: same model, different harness — a second paired audit on SWE-bench
Since posting this proposal I ran a second audit on the same public SWE-bench record, this time on the harness rather than the rank. The question was whether the agent wrapped around a model (tools, prompts, retries) changes its score by more than sampling noise.
What I measured, with hypotheses locked by commit hash before the data pull:
111 pairs of leaderboard submissions that use the exact same model tag. In 68 of them (61 %), the two harnesses differ after Holm correction. The median gap is 8.2 points. Among single-attempt submissions it is 11 of 24.
Kimi K2, an open-weight model, scores anywhere from 43.8 % to 65.4 % depending on the harness.
Even so, the model matters more than the harness: with exact model tags, the harness spread is about 0.4 of the model spread.
The reference harness (mini-SWE-agent, rebuilt from 18,968 per-instance reports) trails the best leaderboard harness by at least 5 points, separably, for 4 of 7 models. This result is fragile: it depends on how submissions that may have seen the repository history are treated.
Code, pre-registration and a small CLI tool, paircheck.py, are in harness_paper/ at github.com/ipezygj/swebench-rank-audit. Given two result files, the tool says whether a claimed improvement is separable and what gap the comparison could have detected at all. It is the first piece of the general tool this grant would fund. The write-up is entered in a Kaggle paper track; it becomes public when the track closes.
What changed for the plan. While writing the limitations section I checked how far the per-instance record goes:
SWE-bench's own per-instance record ends in February 2026, so there is no public per-instance data for the newest models, such as the Claude 5 family.
Epoch AI publishes per-sample logs from its own scaffold up to June 2026, behind a human-check page.
So the plan now has two routes for newer models:
Ask Epoch for a data route to their logs. If they agree, there is no API cost.
For models nobody has published per-instance results for, run them myself on mini-SWE-agent. This costs about $25–$480 per model at current API prices, so roughly ten models fit within the $3,000 minimum.
The goal and minimum stay as they are.
As before, the code, analysis and drafting were done with an AI assistant. I chose the questions, locked the pre-registrations and checked the outputs.