Four-channel leak audit: make pass rates mean capability

A pass rate is a receipt, not a promise. If your AI feature ships because the eval suite passes, you need to know what the model actually did in that room. The room can be rigged by hidden answers, broken tasks, or a grader that disagrees with the prompt.

SWE-Bench Pro exposed hidden solution material to agents and included badly written tasks. The correction revised 102 tasks and removed 17 from the 731-instance public split. On that split, GLM-5.2's pass rate fell from 78.80 percent to 59.51 percent after correction. DeepSeek-V4-Pro's score changed by only 0.05 points between the original and corrected evaluations. That contrast is the warning: a benchmark can measure answer access instead of capability.

Close the leak channels before you trust the number

The SWE-Bench Pro audit named four anti-hacking leakage channels. Run them as an agent eval audit, not a security afterthought. Each check is small, but together they change what the number means.

• Start the task from a fresh repository checkout at the pre-fix commit, with no cached solution files.
• Keep the reference patch, test patch, and generated solution artifacts outside the agent's workspace.
• Filter metadata that names the issue, pull request, repository, or expected fix before the model sees the task.
• Block outbound code hosts during the run, using the same domain blocklist the corrected SWE-Bench Pro environment used: GitHub, GitLab, Gitee, Bitbucket, Codeberg, and GitCode.
• Run a valid alternate fix through the grader and compare the result to the prompt's success criteria.

Fresh checkout sets the baseline. Task images that already contain the fix make the model recover rather than solve. Diff the task snapshot against the base commit before you run the agent, and rebuild the task from the pre-fix state when a solution file appears in the workspace.

Artifact concealment handles tempting files. Patches and generated outputs can sit in the same container as the model. Move them outside the agent's filesystem, not just outside the prompt. Files named after the expected fix become part of the task when the agent can read them.

Metadata filtering catches the less visible leak. Issue titles, labels, file names, and commit messages can encode the intended patch. Redact them before the model sees the task. Clean task manifests should describe the user-visible problem, not the engineering answer.

Network blocking covers the outbound path. Models with outbound access can search public code hosts for the fix. Block the hosts the task does not need. Required package downloads can get a short list of allowed hosts and a request log.

Prompt-to-grader consistency stops reward hacking, where a model games the grader instead of solving the task. Run a correct alternate fix through the grader. A failed correct fix shows the grader is measuring style, not behavior. Add grader tests for valid alternate fixes before you treat the pass rate as a capability signal.

The grader must defend the task

Leak audits are not enough when scoring code is brittle. 75 of the 102 corrected tasks had tests too narrow to accept a correct fix. Brittle graders turn correct fixes into false failures, and loose graders turn broken fixes into false passes. The prompt says make it work, and the grader says make it work exactly this way.

Make the pass rate a regression signal

After the channels are closed and the grader is checked, the pass rate can flag an eval regression. Gains should come from better reasoning, not looser environments. Drops should mean the feature got worse, not that the eval finally stopped leaking.

Keep the audit artifacts with the score. Store the clean task manifest, the network log, and the grader test results next to each pass rate. When a model version changes, compare it against the same clean set. Environment changes require a fresh audit before score comparison.