Open Problems Solved by LLMs? A Survey of Verifiable Mathematical Discovery

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences, Robotics & Autonomous Systems · Depth: Expert, medium

Summary

A survey paper titled "Open Problems Solved by LLMs? A Survey of Verifiable Mathematical Discovery" by Ioannis Tzachristas, Georgios Tzachristas, and Aifen Sui, presented at The Big Picture v2 in July 2026, examines the growing trend of Large Language Models (LLMs) solving previously "open" mathematical problems. The authors analyze how LLMs, typically integrated into search-and-verification loops, advance the state of the art by generating constructions, bounds, or proof certificates. The survey proposes an evidence ladder for evaluating claims of LLM-solved problems, categorizes mathematical subfields by their difficulty for LLM-based discovery, and outlines a timeline of key advancements in verifiable discovery systems. It synthesizes common techniques like tool use, retrieval, search, and verification, emphasizing formal-methods backends such as Linear Temporal Logic (LTL) and Satisfiability Modulo Theories (SMT) solvers for scalable verification. The paper concludes with a reproducibility checklist to enhance trust and facilitate building upon future claims.

Key takeaway

For research scientists exploring LLM capabilities in mathematical discovery, you should adopt a rigorous approach to evaluating "LLM solved" claims. Utilize formal-methods backends like LTL and SMT solvers for robust verification within your LLM-powered systems. Implement the proposed reproducibility checklist to ensure your findings are trustworthy and buildable. This structured methodology will help you differentiate stable evidence from preliminary community reports, fostering more reliable advancements.

Key insights

LLMs, integrated with search and formal verification, are increasingly solving open mathematical problems.

Principles

• "LLM solved" claims require an evidence ladder.
• Mathematical subfields vary in LLM discovery difficulty.
• Verifiable discovery systems rely on specific frameworks.

Method

The paper synthesizes techniques for LLM-based mathematical discovery, including tool use, retrieval, search, and formal verification using LTL and SMT solvers.

In practice

• Use formal-methods backends for verification.
• Apply reproducibility checklists to LLM claims.
• Integrate LLMs into search-and-verification loops.

Topics

• Large Language Models
• Mathematical Discovery
• Formal Verification
• SMT Solvers
• Reproducibility
• Tool Use

Best for: AI Scientist, Research Scientist

Related on AIssential

• Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs — The Mask-Proof pipeline enables scalable, reproducible evaluation of LLM step-level mathematical reasoning using masked-step tasks and an LLM-based judge.
• Benchmarks in Leipzig — LLMs exhibit impressive mathematical reasoning, solving nearly all research-level problems when given iterative attempts.
• VERITAS: Verifier-Guided Proof Search for Zero-Shot Formal Theorem Proving — VERITAS improves LLM theorem proving by integrating all verifier signals into a two-phase, feedback-driven search.
• Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations — LLM robustness to CoT perturbations varies significantly by type and model scale, challenging assumptions that size alone ensures reliability.
• Learning to Reason with Insight for Informal Theorem Proving — Teaching LLMs to identify core techniques significantly improves their informal theorem proving capabilities.
• True Positive Weekly #166 — The issue highlights diverse AI advancements, from genomic challenges to novel LLM behaviors and practical tools.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.