Harnessing Code Agents for Automatic Software Verification

2026-07-08 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

Aria, a novel system for automatic software verification, pairs a general LLM code agent (specifically Claude Opus 4.7) with a robust verification harness to overcome the manual-proof bottleneck in formal verification. This approach achieved full coverage on challenging separation logic proofs, successfully proving all 4,257 lemmas across Iris's four core Coq modules and all 217 lemmas for Rust's standard libraries (Arc, Mutex, RwLock, RefCell) built on Iris. Furthermore, Aria proved all 318 lemmas on the reglang benchmark, significantly outperforming prior LLM provers that achieved only 12% success. It also demonstrated cross-prover generality by proving 72 lemmas on iris-lean, a Lean 4 port of Iris. The system operated autonomously for 380 hours of model time, solving 79.2% of lemmas on the first attempt with a mean of 0.51 retries per lemma.

Key takeaway

For research scientists and software engineers aiming to automate formal verification, this work demonstrates that general LLM code agents, when paired with a robust verification harness, can achieve full proof coverage on complex systems like Iris. You should consider integrating such a harness-driven LLM approach into your verification workflows to significantly reduce manual effort and scale formal methods across challenging software projects.

Key insights

A general LLM code agent, guided by a verification harness, can achieve full, automatic formal software verification.

Principles

• LLMs can generate complex proofs autonomously.
• Verification harnesses ensure proof soundness and completeness.
• Iterative feedback loops improve LLM proof generation.

Method

Aria uses a three-layer architecture: a model layer for proof candidates, an agent layer for orchestration and strategy, and a harness layer for verification and policy enforcement via HHL, driving a feedback-driven retry loop.

In practice

• Use Claude Opus 4.7 for proof generation.
• Implement a declarative Harness Hook Language (HHL).
• Employ a multi-agent loop for extraction, proving, fixing, and polishing.

Topics

• Formal Verification
• Large Language Models
• Coq Proof Assistant
• Iris Separation Logic
• Software Correctness
• Automated Theorem Proving
• RustBelt

Best for: AI Scientist, Software Engineer, Research Scientist

Related on AIssential

• 4 Layer "AI Harness" For LLMs (+54%). Really? — The "AI Harness" improves LLM agent performance by addressing interface mismatches and syntax issues, not by modifying the frozen model.
• LEAP: Supercharging LLMs for Formal Mathematics with Agentic Frameworks — Agentic frameworks like LEAP enable LLMs to achieve state-of-the-art formal theorem proving by bridging informal reasoning with verifiable proofs.
• Components of A Coding Agent — Agentic harnesses significantly enhance LLM coding capabilities by integrating tools, context, and memory management.
• Autoformalization of Agent Instructions into Policy-as-Code — An LLM-driven autoformalization pipeline translates natural language policies into formally verified Cedar policies for enhanced agent safety.
• Using LLMs to Secure Source Code — LLMs streamline vulnerability discovery, shifting the security bottleneck to verification, triage, and patching.
• VeruSAGE: A Study of Agent-Based Verification for Rust Systems — LLMs, when paired with specialized agent systems, can effectively generate correctness proofs for Rust system software.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.