AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection

2026-02-10 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

AnyPoC is a multi-agent framework that addresses the challenge of LLM agents producing plausible but non-functional Proof-of-Concept (PoC) tests or hallucinated traces for bug reports. This system analyzes and fact-checks candidate bug reports, iteratively synthesizes and executes PoCs while collecting traces, and independently re-executes and scrutinizes PoCs to mitigate hallucination and reward hacking. AnyPoC also continuously extracts and evolves a PoC knowledge base for heterogeneous tasks. Applied to 12 critical software systems, including Firefox, Chromium, and OpenSSL, AnyPoC generated 1.3x more valid PoCs for true-positive reports and rejected 9.8x more false-positive reports compared to Claude Code and Codex. It has discovered 122 new bugs (105 confirmed, 86 fixed), with 45 PoCs adopted as official regression tests.

Key takeaway

For software engineering teams evaluating LLM-based bug detection tools, you should integrate automated PoC generation like AnyPoC to move beyond static reports. This framework significantly reduces manual validation effort by generating executable evidence and rejecting false positives, improving the scalability and reliability of your bug-finding pipelines. Consider adopting its multi-agent architecture and self-evolving knowledge base to handle diverse systems and bug types effectively.

Key insights

AnyPoC validates LLM-generated bug reports by autonomously creating and verifying executable Proof-of-Concept tests, significantly reducing false positives.

Principles

• LLM bug reports require executable validation.
• Multi-agent systems enhance validation reliability.
• Independent re-execution mitigates LLM hallucination.

Method

AnyPoC uses analyzer, generator, and checker agents. The analyzer fact-checks reports, the generator synthesizes and executes PoCs, and the checker independently re-executes and scrutinizes them, all while evolving a knowledge base.

In practice

• Integrate PoC generation for LLM bug detection.
• Use multi-agent validation to reduce false positives.
• Adopt self-evolving KBs for project-specific tooling.

Topics

• LLM-based Bug Detection
• Proof-of-Concept Generation
• Multi-Agent Systems
• Software Quality Assurance
• Automated Testing
• Knowledge Base Systems

Code references

• anthropics/claude-code
• google/oss-fuzz
• bytecodealliance/wasmtime
• foundry-rs/foundry
• google/syzkaller

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Software Engineer

Related on AIssential

• Automated Root-Cause Subclassification and No-Code Fix Generation for Invalid Bug Reports — Automating invalid bug report subclassification and no-code fix generation significantly reduces software maintenance overhead.
• TabGuard: Agentic LLM Orchestration for Adaptive Tabular Anomaly Detection via Dynamic Validator Selection and Generation — TabGuard uses agentic LLM orchestration to dynamically select optimal validation strategies for tabular anomaly detection, decoupling reasoning from execution.
• Nobody Is QA Testing Their LLM Apps (That's Going to Be a Problem) — Testing probabilistic AI systems demands a shift from deterministic assertions to statistical quality guarantees.
• Harnessing Code Agents for Automatic Software Verification — A general LLM code agent with a verification harness can fully automate formal software proof generation.
• iCoRe: An Iterative Correlation-Aware Retriever for Bug Reproduction Test Generation — iCoRe enhances bug reproduction test generation via correlation-aware, iterative context retrieval, differentiating code/tests and integrating function call analysis.
• Meta Applies Mutation Testing with LLM to Improve Compliance Coverage — LLMs can significantly improve mutation testing by generating context-aware mutants and targeted tests, enhancing compliance coverage.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.