Manifund foxManifund
Home
Login
About
People
Orgs
Categories
Newsletter
HomeAboutPeopleCategoriesLoginCreate
🍓
🍓
Ilpo

@ipezygj2

0

$0total balance
$0charity balance
$0cash balance

$0 in pending offers

Projects

Leaderboard resolution: an error bar next to every rank on public AI benchmarks

unfunded

By Ilpo1Following1
Leaderboard resolution: an error bar next to every rank on public AI benchmarks
🍓

Ilpo

14 days ago

Update 1: same model, different harness — a second paired audit on SWE-bench

Since posting this proposal I ran a second audit on the same public SWE-bench record, this time on the harness rather than the rank. The question was whether the agent wrapped around a model (tools, prompts, retries) changes its score by more than sampling noise.

What I measured, with hypotheses locked by commit hash before the data pull:

  • 111 pairs of leaderboard submissions that use the exact same model tag. In 68 of them (61 %), the two harnesses differ after Holm correction. The median gap is 8.2 points. Among single-attempt submissions it is 11 of 24.

  • Kimi K2, an open-weight model, scores anywhere from 43.8 % to 65.4 % depending on the harness.

  • Even so, the model matters more than the harness: with exact model tags, the harness spread is about 0.4 of the model spread.

  • The reference harness (mini-SWE-agent, rebuilt from 18,968 per-instance reports) trails the best leaderboard harness by at least 5 points, separably, for 4 of 7 models. This result is fragile: it depends on how submissions that may have seen the repository history are treated.

Code, pre-registration and a small CLI tool, paircheck.py, are in harness_paper/ at github.com/ipezygj/swebench-rank-audit. Given two result files, the tool says whether a claimed improvement is separable and what gap the comparison could have detected at all. It is the first piece of the general tool this grant would fund. The write-up is entered in a Kaggle paper track; it becomes public when the track closes.

What changed for the plan. While writing the limitations section I checked how far the per-instance record goes:

  • SWE-bench's own per-instance record ends in February 2026, so there is no public per-instance data for the newest models, such as the Claude 5 family.

  • Epoch AI publishes per-sample logs from its own scaffold up to June 2026, behind a human-check page.

So the plan now has two routes for newer models:

  1. Ask Epoch for a data route to their logs. If they agree, there is no API cost.

  2. For models nobody has published per-instance results for, run them myself on mini-SWE-agent. This costs about $25–$480 per model at current API prices, so roughly ten models fit within the $3,000 minimum.

The goal and minimum stay as they are.

As before, the code, analysis and drafting were done with an AI assistant. I chose the questions, locked the pre-registrations and checked the outputs.