We compared different LLMs on IMO 2026 [R]

2026-07-26 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Advanced, quick

Summary

A comparison of Large Language Models (LLMs) on International Mathematical Olympiad (IMO) 2026 problems reveals significant performance differences between frontier and sub-frontier models. IMO problems serve as a benchmark due to their novelty, complexity, and demand for general intelligence. Frontier models like "sol" and "fable" achieved perfect or near-perfect scores irrespective of the "harness" or orchestration used. In contrast, models such as "sonnet" and "opus" initially performed poorly, improving with provider harnesses like "claude code" and further with AutoFyn, a custom multi-agent harness. However, even with AutoFyn, these sub-frontier models could not match the top performers. The open-weight model GLM showed similar behavior to sonnet. Grading involved a frontier model and manual verification, which also identified hallucination issues, such as a false solution by sonnet on P3. Notably, the hardest problem, P3, required a key reduction that no sub-frontier model, even with advanced harnesses providing retrieval and verification, was able to identify.

Key takeaway

For AI Engineers evaluating LLMs for complex reasoning tasks, recognize that even advanced multi-agent harnesses like AutoFyn may not bridge the performance gap between sub-frontier and top-tier models on novel, multi-step problems. You should prioritize frontier models for critical applications demanding high accuracy in verifiable domains like mathematics. Additionally, implement robust manual verification steps, as hallucination remains a significant risk even with improved orchestration.

Key insights

Frontier LLMs significantly outperform sub-frontier models on complex, novel math problems, even with advanced orchestration.

Principles

• IMO problems benchmark general intelligence.
• Harnesses improve sub-frontier LLM performance.
• Hallucination persists in verifiable domains.

Method

The study developed AutoFyn, a customizable multi-agent harness, to improve LLM performance on complex math problems by supplying retrieval and verification capabilities.

In practice

• Use frontier models for critical reasoning tasks.
• Implement multi-agent harnesses for sub-frontier LLMs.
• Manually verify LLM outputs in math.

Topics

• Large Language Models
• IMO 2026
• LLM Benchmarking
• Multi-Agent Systems
• AutoFyn
• Model Hallucination

Code references

• SignalPilot-Labs/AutoFyn

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• ASyMOB: Algebraic Symbolic Mathematical Operations Benchmark — LLMs struggle with symbolic math perturbations, indicating pattern memorization, though frontier models show surprising robustness.
• LLM Architecture in 2026: Agent Harnesses, Hybrid Models, and Why Implementation Does Not Lie — Implementation-first analysis of LLM architectures and agent systems reveals evolving training, inference, and context management strategies.
• Am I completely insane for thinking AI is mid — Current LLMs excel at text prediction but fundamentally lack robust reasoning and intuition, leading to limitations and "hallucinations."
• PolyWorkBench: Benchmarking Multilingual Long-Horizon LLM Agents — Multilinguality significantly degrades LLM agent performance in long-horizon tasks, a critical gap in current benchmarks.
• [AINews] OpenAI GPT-next disproves 80 year old Erdős planar unit distance problem for under $1000 — General-purpose LLMs are achieving significant scientific breakthroughs and driving architectural innovation.
• Verifying, Morphing, and Reader-Testing LLMs — LLM verification, hybrid attention, and human-centric evaluation are key to advancing AI capabilities and understanding user perception.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.