Faithful Autoformalization of Natural Language Assertions

2026-06-30 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Software Development & Engineering, Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

Monty is an autoformalization framework designed to synthesize executable assertions from natural-language specifications, bridging the gap between informal developer intent and formal code contracts. This system employs Large Language Models (LLMs) to generate candidate formalizations, which are then filtered and ranked using a novel "clausal coverage" conformance score and validity scores derived from program testing. Evaluated on 541 assertion-generation tasks across 22 Java collection-like classes, Monty demonstrated improved reliability, boosting precision by up to 20 points on average compared to direct LLM translation. For instance, it achieved an 89.7% accuracy with GPT-oss and 75.7% with Qwen2.5-Coder on the C2S-Augmented dataset, with Qwen2.5-Coder's precision rising from 75% to 91.6%. The framework also incorporates active learning for disambiguation when multiple faithful formalizations exist.

Key takeaway

For software engineers or research scientists developing AI-assisted code verification tools, Monty offers a robust approach to autoformalize natural language assertions. You should integrate a multi-stage pipeline that combines LLM candidate generation with execution-level validity checks and semantic conformance scoring. This method significantly improves the precision of formalizations, ensuring greater reliability than naive LLM translation, especially when dealing with ambiguous specifications or testing AI-generated code.

Key insights

Monty reliably converts natural language assertions into executable formal specifications by combining LLM generation with testing-based validation and semantic conformance checks.

Principles

• Autoformalization needs validity and semantic faithfulness.
• Ambiguity requires active learning or inductive biases.
• Roundtrip translation enhances conformance checking.

Method

Monty generates LLM candidates, computes validity via testing and conformance via clausal coverage, then uses a decision matrix and active learning to select the most faithful formalization, handling ambiguity and validity expectations.

In practice

• Use LLMs for initial assertion generation.
• Implement clausal coverage for semantic alignment.
• Employ active learning for ambiguous cases.

Topics

• Autoformalization
• Large Language Models
• Software Verification
• Program Assertions
• Clausal Coverage
• Active Learning

Best for: AI Scientist, Software Engineer, Research Scientist

Related on AIssential

• Untrusted Authors, Trusted Answers: A Calculus of Fidelity-Graded Translations — Trust in multi-language translations stems from architectural checks and graded fidelity, not author provenance.
• Evaluating the Robustness of Proof Autoformalization in Lean 4 — Current LLM-based proof autoformalization models lack robustness to stylistic changes and specific alterations in informal mathematical proofs.
• The Faithfulness Gap: Certifying Semantic Equivalence Between Natural-Language and Formal Mathematical Statements — Certifying semantic equivalence between natural language and formal math statements is crucial for autoformalization faithfulness.
• Automating Quality Assessment with NLP of LLM-Generated Defeaters — Automating defeater quality assessment with NLP and graph analysis reduces subjectivity in safety assurance.
• Logic-Level Evaluation of Logical Table-to-Text Generation — Standard LLM evaluations for logical table-to-text generation are insufficient, masking critical reasoning failures and a meta-logical gap.
• Harnessing Code Agents for Automatic Software Verification — LLM code agents, when unconstrained by fixed strategies and guided by verification harnesses, can fully automate formal software proof generation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.