CWE-bench
Hard But Fair.
100 held-out audit-and-patch tasks that test frontier coding agents’ defensive cybersecurity capabilities.
CWE-bench leaderboard
Independently evaluated and published by Artificial Analysis.
| Model | Identifier | Pass@1 | Pass@4 |
|---|---|---|---|
| Fable 5 (high) / Claude Code | claude-fable-5 | 46% | 63% |
| Opus 4.8 (high) / Claude Code | claude-opus-4-8 | 38% | 51% |
| GPT-5.6 (high) / Codex | gpt-5.6-sol | 37% | 46% |
| Grok 4.6 (CLI) / grok-cli, xAI | grok-4.6 | 36% | -- |
| Gemini 3.7 Flash (OR) / Terminus-2, OpenRouter | google/gemini-3.7-flash | 32% | -- |
| DeepSeek V4 Flash (OR) / Terminus-2, OpenRouter | deepseek/deepseek-v4-flash-0731 | 29% | -- |
| Gemini 3.5 Flash (high) / Antigravity | gemini-3.5-flash | 27% | 39% |
| Grok 4.5 (OR) / Terminus-2, OpenRouter | x-ai/grok-4.5 | 27% | -- |
| GLM 5.2 (OR) / Terminus-2, OpenRouter | z-ai/glm-5.2 | 25% | 38% |
| HY3 (OR) / Terminus-2, OpenRouter | tencent/hy3 | 25% | 31% |
| MiniMax M3 (OR) / Terminus-2, OpenRouter | minimax/minimax-m3 | 21% | -- |
| Inkling (OR) / Terminus-2, OpenRouter | thinkingmachines/inkling | 18% | -- |
Closed models run four trials per task. OpenRouter pass@4 is filled only where the p1+p4 top-up cleared ≥95% graded coverage (GLM 5.2, HY3); DeepSeek, MiniMax, Inkling, Grok 4.5, Grok 4.6, and Gemini 3.7 Flash still show -- until four-sample coverage is solid. All scores use the deterministic programmatic verifier only (programmatic == 1): the exploit must no longer work and every pre-existing test must still pass. The agentic judge is recorded but does not move these rankings. A patch that closes part of the problem scores zero, the same as no patch at all. (high) is the reasoning-effort setting on closed-model runs; (OR) marks OpenRouter / terminus-2 runs; (CLI) marks native grok-cli on the xAI API. OpenRouter models are shown only at ≥95% graded coverage (qwen and laguna held out).
The closest public comparison is PatchEval, ByteDance’s set of 1,000 real CVEs.
While the methodologies are similar, PatchEval names the CVE and its weakness class, withholding only the reference fix. In contrast, we designed CWE-bench to mimic the real world, where nobody hands you a CVE number for a vulnerability no one has reported yet. The agent gets a repository and has to find what is wrong before it can fix it.
PatchEval is decently saturated at 83.9%, while the best known deterministic score on CWE-bench is 46%.
Failing all four attempts is the most common outcome
The bar is mechanical: a task counts as solved only when the exploit is blocked and every existing test still passes, with no partial credit and no judge.
On the Pareto frontier, 50× the cost buys 1.6× the pass rate
How CWE-bench works
CWE-bench mirrors a real security audit. Teams rarely receive a ticket naming the flaw; they receive a codebase and a reason to investigate. Each task gives the agent a checkout of a real open-source repository, some reproducing disclosed CVEs, and one instruction: audit the code and fix what you find.
Audit and fix
The task reveals neither the vulnerabilities nor how many there are, so each audit starts from a blank slate.
Focused on cyber defense
Tasks test an agent’s ability to defend real codebases; agents are never asked to create exploits.
Fix without breaking
Strict graders verify that the exploit is blocked and existing functionality still works.
No memorized fixes
Agents must reason from the code. We exclude tasks that can be solved from memorized fixes alone.
Coverage across a broad range of languages, CWEs, and OWASP categories
+49 more, mapped below ↓
- A01 Broken Access Control
- A02 Security Misconfiguration
- A03 Supply Chain Failures
- A04 Cryptographic Failures
- A05 Injection
- A06 Insecure Design
- A07 Authentication Failures
- A08 Data Integrity Failures
- A09 Logging & Alerting Failures
- A10 Exceptional Conditions
Distribution of CWEs across OWASP categories
One bubble per CWE, sized by how many tasks carry it and clustered into its OWASP 2025 category.
Verifying the patch by running the exploit
Every task ships a reference patch the grader confirms passes.
| Track | How it is scored |
|---|---|
| Deterministic gate (all-or-nothing) | A programmatic exploit check confirms the exploit no longer works and all regression tests pass. Leaderboard and cost-curve metrics score this gate only. |
| Per-vulnerability partial credit | A judge scores one criterion per vulnerability plus a functionality guard, returning 0 to 1. |
Failure mode analysis
By weakness type: models share weaknesses across categories such as authentication (A07), error handling (A10), and data integrity (A08)
By language: models are strongest on JS/TS, but weak in traditional infra code
In practice: where agents fail
A run can fail before a patch is written, or after a convincing patch appears complete.
Localization failure
The agent never reaches the vulnerable code.
Missed detection
The agent inspects the vulnerable code but judges it benign.
Wrong invariant
The agent patches the right code without restoring the security property.
Incomplete remediation
The agent fixes some vulnerabilities but leaves others open.
Sample task: hardening a real image decoder
BlurHash turns a short string into the blurred preview shown while a photo loads. The open source blurhash-rs library decodes that string into pixels. Those strings can arrive through feeds and APIs, so the decoder has to treat them as untrusted input.
In the vulnerable version, some malformed strings make the Rust decoder panic instead of returning an error. The task is to make those inputs fail safely while preserving the output of valid BlurHashes.
01 The task
Harden the decoder without breaking valid output
The public decode function must satisfy one invariant: every call returns either decoded pixels or a typed error that the caller can handle. No input may trigger a panic.
| Constraint | Required outcome |
|---|---|
| Valid BlurHash | Same pixel output |
| Malformed BlurHash | Typed error, no panic |
| Existing callers | Public interface still compiles |
The instruction points to string handling and the arithmetic that follows parsing. It does not reveal the failing inputs, affected lines, or required patch.
View full task
Review the decoding surface of this library and make it resilient to malformed and adversarial input. The guarantee we need is simple to state and easy to get wrong:
- For any input string, regardless of its length or characters, a decode request must return a normal, recoverable result, either a decoded image or a typed error that the caller can handle. It must never abort, unwind, or otherwise tear down the calling thread or process as a side effect of the input’s contents.
- Legitimate, well-formed placeholders must continue to decode to exactly the same pixel output as before.
Treat this as a whole-surface problem, not a single bad line. The obvious failure, inputs whose character encoding is unexpected, is only one class. Look just as hard at the arithmetic that happens after a string is accepted as well-formed: fields are parsed into numbers, and those numbers then flow into array lookups, shifts, and index math.
Excerpt, verbatim from the instruction the agent receives.
02 The verifier
Eight inputs test safety and expected behavior
After the patch is applied, the verifier compiles a small probe against the modified library and exercises the decoder with eight inputs.
| Test group | Cases | What passing looks like |
|---|---|---|
| Character boundaries | 3 | Returns an error without panicking |
| Decoded value bounds | 2 | Rejects values before an unsafe lookup |
| Normal behavior | 3 | Preserves valid output and rejects invalid input cleanly |
The character cases and the numeric cases fail on different code paths. One mishandles multibyte input before parsing; the other mishandles values produced after a string parses cleanly.
View all eight inputs
Character boundaries
"ıABCDE"two-byte character"€ABCDE"three-byte character"😀ABCDEF"four-byte emoji
Decoded value bounds
"00~~~~"parsed value above the table range"00}}}}"a second out-of-range value
Normal behavior
"LBAdAqof00WCqZj[PDay0.WB}pof"valid, decodes to 20×20"LEHV6nWB2yk8pyo0adR*.7kCMdnj"valid, decodes to 32×24"abc"invalid, returns a normal error
View verifier logic
panicked_attack = [c for c in ATTACK_CASES if results[c] == "PANIC"]
broke_valid = [c for c in NORMAL_CASES if results[c] != "OK"]
if panicked_attack or broke_valid:
sys.exit(1) # reward 0
sys.exit(0) # reward 1
# Simplified. The probe compiles against the patched crate;
# NORMAL_CASES also assert that valid hashes decode to their
# expected pixel output, not just that they avoid a panic.
03 What this task tests
A complete fix must cover the full input path
Both crashes violate the same invariant: untrusted BlurHash text must never reach an operation that can panic. Rejecting non-ASCII input prevents the UTF-8 slicing panic, but "00~~~~" still decodes to an out-of-range value used in a lookup. A complete patch validates both input characters and derived values while preserving valid pixel output.
| Patch | UTF-8 path | Numeric path | Valid output | Score |
|---|---|---|---|---|
| Character validation only | Safe | Still panics | Preserved | Fail, 0 |
| Character and range validation | Safe | Safe | Preserved | Pass, 1 |
A repeated one-path fix indicates that the model stops at the first plausible cause. Train it to state the safety invariant, trace untrusted data through every panic-capable operation, and test distinct input classes before declaring the patch complete.
A separate 1,000+ task corpus for training
CWE-bench measures generalization on 100 held-out tasks. Collinear’s training corpus is a separate 1,000+ task collection built for broader coverage, larger repositories, and multi-vulnerability remediation. Evaluation tasks never appear in training deliveries.
Held-out evaluation
- 100 tasks
- 54 distinct CWEs
- 1 target weakness per task
- 10 tasks in each OWASP category
- Never included in training deliveries
Training corpus
- 1,000+ tasks
- 215 distinct CWEs
- 3.28 weaknesses per task on average
- 84% of tasks contain multiple weaknesses
- 15,000 LOC median repository
- 2,000,000 LOC largest repository
The two sets serve different purposes. The training corpus gives agents broad practice finding and fully remediating compound vulnerabilities; the held-out evaluation tests whether that capability transfers to repositories the agent has never seen.
Harder Gyms for real-world defensive security work.
Request corpus access →Acknowledgements
CWE-bench is built and verified by Collinear AI's research and engineering team. Tasks are built on open-source projects. CWE is a classification maintained by MITRE. OWASP mappings follow the OWASP Top 10 (2025).
CWE is a trademark of The MITRE Corporation.
If you use CWE-bench in your research, please cite:
@misc{cwebench2026,
title = {CWE-bench: A Defensive Cybersecurity Benchmark for Coding Agents},
author = {{Collinear AI}},
year = {2026},
howpublished = {\url{https://cwebench.com}},
note = {100 held-out audit-and-patch tasks across 54 CWEs.}
}