Verifier-First Evaluation of Agentic LLMs for Infrastructure-as-Code Generation

2026-07-24 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Expert, extended

Summary

A verifier-first empirical study evaluated seven agentic strategies for Terraform Infrastructure-as-Code (IaC) generation on IaC-Eval v2, a modernized 186-task AWS/Terraform benchmark with Rego v1 policies. The research found that active retrieval via ReAct agents significantly improved Qwen2.5-Coder 7B's pass@1 rate from 14.0% to 45.7% ($p<0.0001$), primarily by reducing VALIDATE_FAIL errors from 144 to 66 tasks. Iterative refinement with verifier feedback achieved 62.9% for Qwen 7B and 84.4% for GPT-4o, demonstrating binary convergence in error resolution. Furthermore, GEPA reflective instruction optimization enhanced the Active RAG baseline by +7.5 percentage points ($p=0.026$), proving prompt optimizers can improve verifiable IaC generation. The study also revealed that 79% of post-refinement OPA failures were information gaps, resolvable by making policy text visible, and highlighted that instruction-level optimization (GEPA) surpassed demonstration-level optimization (SIMBA) for schema-constrained code generation tasks.

Key takeaway

For AI Engineers building agentic LLMs for Infrastructure-as-Code, you should adopt a verifier-first evaluation approach to diagnose specific failure stages. Prioritize active retrieval to mitigate schema validation errors and implement iterative refinement for repairable issues. Crucially, integrate policy text directly into prompts to resolve policy compliance failures, as most are information gaps, not model capability limitations. Consider GEPA for instruction optimization to enhance schema-constrained code generation.

Key insights

IaC generation requires stage-aware evaluation and intervention, as retrieval, repair, and optimization target distinct failure types.

Principles

• IaC evaluation needs stage-specific failure decomposition.
• Retrieval and repair address orthogonal failure classes.
• Instruction optimization outperforms demonstration optimization for schema constraints.

Method

The study used a three-stage verifier-first pipeline (terraform validate, terraform plan, opa eval) to classify failures and applied McNemar's test for statistical validation of agentic strategies.

In practice

• Use active retrieval to reduce schema validation errors.
• Implement iterative refinement for recoverable validation/planning failures.
• Employ GEPA for instruction optimization in schema-constrained code.

Topics

• Infrastructure-as-Code
• Agentic LLMs
• Terraform Generation
• Retrieval-Augmented Generation
• Prompt Optimization
• Verifier-First Evaluation

Best for: AI Architect, Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Evaluating Agentic Configuration Repair for Computer Networks — Agentic LLM architectures significantly improve network configuration repair efficacy and safety through iterative validation.
• LLM-as-a-Verifier: A General-Purpose Verification Framework — LLM-as-a-Verifier uses scoring token logits to provide fine-grained, scalable verification for agentic tasks without retraining.
• ASMR: Agentic Schema Generation for Ship Maintenance Report Writing — Agentic AI and reinforcement learning can automate schema generation from unstructured text.
• Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios — LLM agents struggle with end-to-end software generation, often preferring monolithic designs over modularity.
• An LLM-Driven Workflow for Automated Process Control Strategy Generation and Tuning from Dynamic Process Models — LLMs can automate multi-variable control design and tuning through structured code generation and validation.
• Agentic Engineering and the Lost Art of Verification — Advanced AI agents enable scalable software development, automated verification, and personalized knowledge management, shifting focus from manual coding to oversight.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.