Agentic Synthesis against Counterexample-Supplemented Sketches
Summary
Agentic Synthesis against Counterexample-Supplemented Sketches is a repository-native method designed for systems where governing policy evolves during implementation, addressing the issue of coding agents fixing examples without capturing underlying domain rules. A human initiates with a partial code sketch, and a coding agent generates the initial implementation. Upon concrete failure exposing missing policy, an operator explicitly approves the corrected behavior and rule. The agent then revises the sketch and repairs or regenerates code and prompt surfaces for that specific counterexample. Demonstrated with CatSynth and GPT-5.4-mini, one open-world run showed 8 of 14 candidate cases became counterexamples. Rebuilding from the evolved sketch passed 19 of 21 withheld cases, outperforming rebuilding from the initial sketch (15 of 21). Retaining code required 9 Developer calls and 719 lines of churn, significantly less than replay-all's 15 calls and 2,394 lines, passing 18 of 21 withheld cases.
Key takeaway
For AI Engineers building agentic coding systems where policy evolves, you should implement a counterexample-supplemented sketch workflow. This means requiring explicit operator approval for policy-changing failures, ensuring every accepted counterexample revises the governing sketch, not just the code. Separate replay and semantic compare checks will clarify whether failures are state-repair or policy violations, making your agentic repairs durable and inspectable against a curated regression set.
Key insights
Explicit operator approval of counterexamples drives sketch evolution, capturing policy for robust agentic code synthesis.
Principles
- Accepted CEs must update the sketch with reviewed rules.
- Operator-approved outputs are authoritative, not model explanations.
- Code should be regenerable from evolved sketch and anchors.
Method
A human provides a partial sketch. An agent implements. Failures become operator-approved counterexamples, revising the sketch. The agent repairs code/prompts, and a gate validates against regressions.
In practice
- Begin with a short sketch defining operations and priorities.
- Collect examples that distinguish plausible but wrong repairs.
- Separate replay and semantic compare checks for clarity.
Topics
- Agentic Coding
- Program Synthesis
- Policy Learning
- Counterexample-Guided Synthesis
- Regression Testing
- LLM Software Engineering
Code references
Best for: Research Scientist, Machine Learning Engineer, AI Scientist, AI Engineer, MLOps Engineer
Related on AIssential
See Counsel's argued verdicts on the open AI decisions leaders are weighing →
Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.