Evaluation of LLMs for Mathematical Formalization in Lean

2026-06-06 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, extended

Summary

This study evaluates various Large Language Models (LLMs) for formal mathematical proof generation within the Lean 4 theorem proving system. Researchers compared models using pass@k and refine@k metrics on 50-problem subsets of the miniF2F and miniCTX datasets. Results indicate Gemini 3.1 Pro and Claude Opus 4.7 are top performers, with Gemini 3.1 Pro achieving 92% success on miniF2F via refine@32 and Opus 4.7 reaching 86% on miniCTX via refine@32. Considering cost, NVIDIA Nemotron 3 Super and GPT-OSS 120B proved most efficient, costing less than \$0.01 per correct proof. The analysis also found that iterative refinement strategies generally improved performance by an average of +3.14% across models, highlighting the value of feedback loops.

Key takeaway

For research scientists developing formal proof generation systems, you should prioritize LLMs like Gemini 3.1 Pro or Claude Opus 4.7 for their superior accuracy in Lean 4. If budget is a constraint, consider NVIDIA Nemotron 3 Super or GPT-OSS 120B, which offer competitive performance at significantly lower costs. Implement iterative refinement strategies, as they consistently improve proof success by leveraging compiler feedback.

Key insights

Iterative refinement significantly boosts LLM performance in formal mathematical proof generation.

Principles

• Formal proof generation benchmarks LLM reasoning.
• Cost-efficiency varies widely among LLMs.
• Iterative feedback improves proof success.

Method

The study used a standardized zero-shot prompting paradigm with a temperature of 0.5. It employed pass@k (32 independent generations) and refine@k (up to 32 iterative refinements with Lean verifier feedback) for evaluation.

In practice

• Prioritize Gemini 3.1 Pro or Claude Opus 4.7 for high accuracy.
• Use Nemotron or GPT-OSS for cost-effective Lean 4 proofs.
• Implement iterative refinement for better proof success rates.

Topics

• Large Language Models
• Formal Proof Generation
• Lean 4
• Theorem Proving
• Model Evaluation
• Cost Efficiency

Code references

• uw-math-ai/LLMsLean

Best for: AI Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Evaluation of LLMs for Mathematical Formalization in Lean — LLMs like Gemini 3.1 Pro and Claude Opus 4.7 show high effectiveness in generating formal Lean 4 proofs, with some models offering cost efficiency.
• Benchmarks in Leipzig — LLMs exhibit impressive mathematical reasoning, solving nearly all research-level problems when given iterative attempts.
• Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming — LLMs can actively verify and debug competitive programming code, but current inference costs limit practical deployment.
• Recent LLMs can use filler tokens or problem repeats to improve (no-CoT) math performance — Recent LLMs can use filler tokens or problem repeats to boost math performance without explicit Chain-of-Thought.
• TheoremBench: Evaluating LLMs on Theorem Proving in Formal Mathematics — TheoremBench evaluates LLMs on formal math proving, revealing performance gains with explicit premises and current prover inefficiencies.
• GPT 5.5 vs Opus 4.8 vs Gemini 3.5 - Which Model Should You Use? — Optimal AI model selection requires matching specific model strengths to task requirements, as no single model excels universally.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.