Update (Nov 2026): HumanEval Pro / MBPP Pro findings are public
The 14-day disclosure window for the CodeEval-Pro authors has closed without a reply. We've now published the full audit: 40/164 HumanEval Pro and 25/378 MBPP Pro tasks accept provably wrong code. On top of that, 3 MBPP Pro tests fail their own reference solution (exact float equality), and one BigCodeBench-Lite Pro test only passes in UTC+8.
In total there are 191 confirmed weak-test tasks across 13 suites, each with a witness input, a diff and a repro command. Code is MIT, data is CC BY. Repo and leaderboard: https://github.com/BliZiy/shuna-audit. Public issue: https://github.com/CodeEval-Pro/CodeEval-Pro/issues/5.
Measured impact on 6 open models: up to 3.5 points of score inflation (largest on MBPP; under 2 points elsewhere).
Next: a 50-task SWE-bench Verified sample (mutated gold patches vs. FAIL_TO_PASS tests), then LiveCodeBench and CRUXEval. Funding still goes first to API, compute and independent expert review of borderline items. Caveats as always: small samples, and "confirmed" means our oracle/manual review, not independent experts. Feedback and disputes are welcome. — BliZi