Manifund foxManifund
Home
Login
About
People
Orgs
Categories
Newsletter
HomeAboutPeopleCategoriesLoginCreate
🥭
🥭
BliZi

@shuna-audit

0

$0total balance
$0charity balance
$0cash balance

$0 in pending offers

Projects

Shuna Audit: an open leaderboard of grader quality for AI coding benchmarks

unfunded

By BliZi1Following1
Shuna Audit: an open leaderboard of grader quality for AI coding benchmarks
🥭

BliZi

6 days ago

Update (Nov 2026): HumanEval Pro / MBPP Pro findings are public

  • The 14-day disclosure window for the CodeEval-Pro authors has closed without a reply. We've now published the full audit: 40/164 HumanEval Pro and 25/378 MBPP Pro tasks accept provably wrong code. On top of that, 3 MBPP Pro tests fail their own reference solution (exact float equality), and one BigCodeBench-Lite Pro test only passes in UTC+8.

  • In total there are 191 confirmed weak-test tasks across 13 suites, each with a witness input, a diff and a repro command. Code is MIT, data is CC BY. Repo and leaderboard: https://github.com/BliZiy/shuna-audit. Public issue: https://github.com/CodeEval-Pro/CodeEval-Pro/issues/5.

  • Measured impact on 6 open models: up to 3.5 points of score inflation (largest on MBPP; under 2 points elsewhere).

Next: a 50-task SWE-bench Verified sample (mutated gold patches vs. FAIL_TO_PASS tests), then LiveCodeBench and CRUXEval. Funding still goes first to API, compute and independent expert review of borderline items. Caveats as always: small samples, and "confirmed" means our oracle/manual review, not independent experts. Feedback and disputes are welcome. — BliZi