Measuring LLMs' Ability to Develop Exploits

2026-05-21 · Source: Anthropic Frontier Red Team Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Expert, long

Summary

Anthropic's Claude Mythos Preview model demonstrates a significant advancement in exploit development capabilities, outperforming previous frontier models. Released carefully through Project Glasswing, Mythos Preview can identify complex vulnerabilities, create exploit primitives, and combine them into full attack chains. New quantitative benchmarks, ExploitBench and ExploitGym, were developed to precisely measure these capabilities, as existing public benchmarks were insufficient. On ExploitBench, focusing on 41 V8 engine vulnerabilities, Mythos Preview achieved arbitrary code execution (ACE) on 21 CVEs, while no other model achieved even one. ExploitGym, covering 898 vulnerabilities across OSS-Fuzz, V8, and Linux kernel, saw Mythos Preview achieve 157 intended vulnerability exploits within two hours, significantly more than Opus 4.6's 15. Furthermore, an updated SCONE-bench for smart contract exploitation showed Mythos Preview exploiting \$35 million worth of contracts, a 75% increase over the next best model, and exploiting every tested vulnerability. This performance suggests exploit development expertise will become commoditized.

Key takeaway

For AI Security Engineers assessing LLM-driven threats, Claude Mythos Preview's advanced exploit capabilities mean you must re-evaluate your defense strategies. This model can achieve arbitrary code execution and exploit smart contracts worth millions, indicating a rapid commoditization of exploit development. You should prioritize developing more sophisticated, LLM-aware security measures and actively engage with new, high-quality benchmarks like ExploitBench and ExploitGym to understand evolving risks.

Key insights

Claude Mythos Preview significantly advances LLM exploit development, achieving end-to-end attacks and outperforming prior models on new benchmarks.

Principles

• Exploit development requires a capability ladder.
• Quantitative benchmarks are crucial for LLM evaluation.
• LLM exploit capabilities are rapidly accelerating.

Method

ExploitBench decomposes exploit development into 16 programmatically verified capabilities across five tiers, testing against V8 engine vulnerabilities with a 300-turn budget and adaptive prompting.

In practice

• Use ExploitBench for fine-grained exploit capability analysis.
• Evaluate LLMs against V8, OSS-Fuzz, and Linux kernel targets.
• Test smart contract exploitation with SCONE-bench.

Topics

• LLM Exploit Development
• Claude Mythos Preview
• ExploitBench
• ExploitGym
• Smart Contract Exploitation
• V8 Engine Vulnerabilities
• Cybersecurity Benchmarking

Code references

• google/oss-fuzz
• SunWeb3Sec/DeFiHackLabs
• anthropics/scone-bench

Best for: CTO, VP of Engineering/Data, Research Scientist, AI Security Engineer, AI Scientist, Director of AI/ML

Related on AIssential

• Assessing Claude Mythos Preview’s cybersecurity capabilities — Advanced language models like Claude Mythos Preview exhibit unprecedented autonomous vulnerability discovery and exploitation capabilities across diverse software.
• CyberChainBench: Can AI Agents Secure Smart Contracts Against Real-World On-Chain Vulnerabilities? — LLM agents show promise but significant limitations in smart contract security tasks, especially patching.
• Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuning — Data quality and structured evaluation are paramount for effective LLM-based exploit generation.
• An initiative to secure the world's software | Project Glasswing — Advanced LLMs trained for code can autonomously identify and chain complex software vulnerabilities, enhancing cybersecurity defense.
• Reverse engineering Claude's CVE-2026-2796 exploit — Claude Opus 4.6 demonstrated autonomous browser exploit generation, signaling advancing LLM cyber capabilities.
• New benchmark shows Claude Mythos and GPT-5.5 can develop real browser exploits autonomously — AI agents like Claude Mythos can autonomously develop browser exploits, demonstrating advanced cybersecurity capabilities.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Anthropic Frontier Red Team Blog.