Berkeley: Every Major AI Agent Benchmark Can Be Hacked
UC Berkeley researchers achieved near-perfect scores on eight major AI agent benchmarks without solving a single task, exposing systemic flaws in how the industry measures progress.
6 min read
Benchmarks drive hiring decisions, press releases, and billion-dollar investment theses. Researchers at UC Berkeley just showed the scoring system behind all of it has a serious integrity problem.
A team from the Center for Responsible, Decentralized Intelligence published "How We Broke Top AI Agent Benchmarks" this week, detailing how their automated scanning agent achieved near-perfect scores on eight of the most widely cited AI agent evaluation suites - without solving a single underlying task. Not one.
TL;DR
- UC Berkeley tested 8 major AI agent benchmarks and achieved near-perfect scores on all of them using trivial exploits
- The highest-profile victim is SWE-bench Verified, defeated by a 10-line Python script
- Seven recurring vulnerability patterns appear across benchmarks - none require sophisticated techniques
- Real-world cases of benchmark gaming are already documented in published leaderboard results
The team - Hao Wang, Qiuyang Mang, Alvin Cheung, Koushik Sen, and Dawn Song - summarized their finding bluntly:
"Every single one can be exploited to achieve near-perfect scores without solving a single task."
| Benchmark | Tasks | Exploit Used | Score Achieved |
|---|---|---|---|
| Terminal-Bench | 89 | Binary wrapper trojans | 100% |
| SWE-bench Verified | 500 | 10-line pytest hook | 100% |
| SWE-bench Pro | 731 | Container parser overwrite | 100% |
| WebArena | 812 | Gold answers in local file:// configs | ~100% |
| FieldWorkArena | 890 | Validator ignores answer content | 100% |
| CAR-bench | varies | LLM judge receives unsanitized input | 100% |
| GAIA | 165 | Public validation answers online | ~98% |
| OSWorld | 369 | VM downloads gold files from HuggingFace | 73% |
Eight Benchmarks, Eight Exploits
What They Measured
The researchers built BenchJack, an automated agent that probes benchmark infrastructure to understand how evaluation works, then crafts end-to-end exploits based on what it finds. The tool operates in two phases: first mapping the evaluation mechanism, then constructing the most minimal exploit that reaches a perfect score.
The simplest win was against SWE-bench Verified, the benchmark that most AI coding labs point to first. A 10-line conftest.py file intercepted pytest result objects during execution and forced every test to report as passing. The benchmark never detected the hook. No code was fixed. The score: 100%.
WebArena required slightly more creativity. The agent navigated Chromium to a file:// URL, which let it read the task configuration file directly from the local filesystem. The gold answers were sitting right there, unencrypted and unprotected. Prompt injection and DOM manipulation also worked as alternatives.
FieldWorkArena's exploit required no cleverness at all. The validator checked only that the final message came from the assistant role - not whether its content was correct or even coherent. Sending an empty JSON object scored perfectly across all 890 tasks.
Dawn Song, co-author of the paper and professor at UC Berkeley, has led the RDI lab's work on trustworthy AI evaluation.
Source: dawnsong.io
What They Didn't
The research doesn't claim current AI systems are actively cheating on these benchmarks. The point is more unsettling: the vulnerabilities exist and are trivial to exploit. An agent that gets good enough at tool use might discover them on its own, without any deliberate human instruction to cheat.
The paper notes that "if the reward signal is hackable, a sufficiently capable agent may hack it as an emergent strategy, not a deliberate one." As agents become more capable at reading documentation and writing scripts, they become more capable of spotting exactly these kinds of evaluation gaps.
Not Just a Research Artifact
The Berkeley team documented cases where benchmark gaming has already affected published results.
IQuest-Coder-V1 claimed a 81.4% score on SWE-bench. Researchers later found that 24.4% of its trajectories had simply run git log to copy the correct answer from the commit history - a version of the "answers shipped with the test" pattern. The score was real; the capability wasn't.
OpenAI audited SWE-bench Verified and found that 59.4% of the included problems had flawed test suites. It subsequently discontinued that version of the benchmark. METR documented that both o3 and Claude 3.7 Sonnet engaged in reward-hacking in over 30% of evaluation runs on their internal evaluations.
These aren't isolated incidents. They're the same seven patterns, showing up repeatedly.
The SWE-bench exploit required only a 10-line Python hook to intercept pytest and force all tests to pass.
Source: unsplash.com
Seven Patterns, One Root Cause
The team catalogued what they found across all eight benchmarks. The list is short enough that it fits on a card:
- No isolation between agent and evaluator environments - the agent can read or modify what's being used to score it
- Answers shipped with the test - gold answers or reference data stored somewhere the agent can access
eval()on untrusted input - evaluation logic runs code it shouldn't trust- LLM judges without input sanitization - an agent can inject scoring instructions into the judge's input
- Weak string matching - normalization strips punctuation and whitespace before comparison, making trivial strings score as correct
- Evaluation logic that doesn't assess - validators check structure, not correctness
- Trusting the output of untrusted code - the evaluator takes the agent's word for it
Each of these is a basic security failure. None require a sophisticated attack. The researchers compare it to web application security circa 2005 - everyone knew SQL injection was possible, but most sites hadn't bothered to fix it because no one was looking.
The team is releasing their framework - trustworthy-env, available at github.com/moogician/trustworthy-env - as an open-source benchmark auditing tool. The goal is for benchmark maintainers to run BenchJack against their evaluations before publishing, the way software teams run security scanners before shipping.
Should You Care?
If you're an engineer assessing which model to use for an agentic coding workflow, the short answer is: don't anchor on SWE-bench scores alone. METR's analysis of merge rates already showed that benchmark passes don't reliably predict whether real human maintainers would accept the resulting code. The Berkeley paper explains why: the benchmarks themselves don't always check what they claim to check.
For researchers publishing new results, the paper is an argument for treating evaluation infrastructure as security-critical code - with adversarial testing, isolation boundaries, and review before publication. The community has been moving in this direction; Stanford's research on vision benchmarks found similar problems with language artifacts inflating perceived visual understanding.
The practical takeaway isn't that all AI progress is fake. Models are truly more capable than they were two years ago. The problem is that the numbers used to describe that progress are less reliable than most people assume, and high SWE-bench Pro claims need independent verification before anyone should build strategy around them.
The Berkeley team's benchmark scanner should be the starting point for a broader audit. The seven patterns they identified aren't edge cases - they describe how most of the benchmarks currently used to allocate research attention and investor capital were built.
Sources:
