Patterns for Building Cybersecurity Evals

2026-06-21 · Source: Eugene Yan · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy, Robotics & Autonomous Systems · Depth: Advanced, long

Summary

The article details common patterns and benchmarks for evaluating AI models' cybersecurity capabilities, focusing on their ability to find and exploit vulnerabilities. Evaluations typically feature a sandboxed target, variable difficulty inputs (e.g., zero-day/one-day scenarios), specialized tools, and a deterministic grader with partial credit via subtasks. Benchmarks discussed include Cybench, measuring capture-the-flag performance (top models achieved 17.5% success); CVE-Bench, evaluating 40 critical NVD vulnerabilities (top agents reached 12.5% success); CyberGym, assessing Proof of Concept generation for memory-safety flaws (top models achieved 22.0% success); ExploitGym, focusing on full code execution (top models achieved 157 exploits, reduced with defenses); ExploitBench, evaluating V8 JavaScript engine bugs (a research model achieved full code execution on 18 of 41 bugs); MHBench, assessing multi-host red-teaming (a new system boosted success from 3 to 37 of 40 networks); and SCONE-Bench, measuring smart contract exploitation (models generated 207 exploits, simulating \$550 million stolen).

Key takeaway

For AI Security Engineers developing or deploying AI agents for offensive or defensive cybersecurity, you should prioritize structured evaluation environments that incorporate partial credit for subtasks. This approach provides granular insights into agent capabilities beyond simple pass/fail outcomes, revealing specific areas for improvement. Furthermore, invest in robust system frameworks, as they significantly impact agent performance more than the underlying model alone, especially when facing real-world security defenses.

Key insights

AI agent cybersecurity evaluations require structured environments and granular scoring to assess capabilities and limitations.

Principles

• Outcome-based grading is common for open-ended exploitation.
• Partial credit via subtasks provides granular progress insight.
• System framework often outweighs the underlying model's performance.

Method

Build evals with a sandboxed target, variable inputs for difficulty, agent tools, and a deterministic grader that awards partial credit for subtasks.

In practice

• Use subtasks to track agent progress along attack chains.
• Test agents in both zero-day and one-day vulnerability scenarios.
• Implement robust system frameworks to enhance agent performance.

Topics

• AI Agent Evaluation
• Cybersecurity Benchmarks
• Vulnerability Exploitation
• Smart Contract Security
• Red Teaming
• Memory Safety

Code references

• google/oss-fuzz

Best for: AI Engineer, Research Scientist, CTO, AI Scientist, AI Security Engineer, Machine Learning Engineer

Related on AIssential

• TAI #204: Are AI Agents Starting A Cybersecurity Arms Race? — AI agents are rapidly transforming cybersecurity, enabling both advanced attacks and unprecedented defensive capabilities.
• When Skills Don't Help: A Negative Result on Procedural Knowledge for Tool-Grounded Agents in Offensive Cybersecurity — Agent Skills' benefit diminishes significantly when environment feedback is high, particularly in offensive cybersecurity.
• A Security Analysis of Long-Horizon Agentic AI Systems: Threats, Evaluation, and Framework Development — The paper analyzes security threats and proposes a taxonomy and attack propagation framework for long-horizon agentic AI systems.
• Stanford and Harvard just dropped the most disturbing AI paper of the year — Incentivized AI agents will discover and exploit manipulative behaviors, revealing critical security and governance gaps.
• What cybersecurity pros need to know about OpenClaw and Moltbook — Locally run AI agents introduce significant cybersecurity risks due to high permissions and user configuration challenges.
• Eval awareness in Claude Opus 4.6’s BrowseComp performance — Advanced models can detect evaluations, identify benchmarks, and decrypt answer keys, challenging static benchmark reliability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Eugene Yan.