Too long; didn’t solve

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

A study by Lucía Cabrera, Jocelyn D'Arcy, and Isaac Saxton-Knight, presented at EvalEval 2026, investigates how structural properties influence large language model performance on mathematical benchmarks. The research specifically examines prompt length and solution length, analyzing their relationship to model failure on a newly constructed adversarial dataset of expert-authored mathematics problems. Across five evaluated models, the authors found that both prompt length and solution length are positively, though modestly and descriptively, associated with increased model failure. A secondary, exploratory analysis of cross-model disagreement was also included, interpreted cautiously due to mechanical constraints. The main finding highlights that structural length is linked to empirical difficulty in these benchmarks, suggesting it should be considered a potential confounder in mathematical model evaluations.

Key takeaway

For AI Scientists evaluating large language model reasoning on mathematical benchmarks, you must account for problem length. This study demonstrates that both prompt and solution lengths are positively associated with increased model failure, suggesting length acts as a significant confounder. You should design evaluations that control for or explicitly analyze these structural properties to ensure accurate assessment of true reasoning capabilities, rather than just length-handling capacity.

Key insights

Increased prompt and solution lengths are positively associated with large language model failure on mathematical benchmarks.

Principles

• Structural length correlates with LLM failure rates.
• Problem length acts as a confounder in LLM math evaluations.

Method

Investigated prompt and solution length variables using a new adversarial dataset of expert-authored math problems across five LLMs to analyze performance.

In practice

• Account for problem length when designing LLM math benchmarks.
• Analyze length variables as potential confounders in evaluation results.

Topics

• Large Language Models
• Mathematical Reasoning
• LLM Evaluation
• Prompt Length
• Solution Length
• Adversarial Benchmarks

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Finding the Optimal Reasoning Budget for LLMs — Optimal LLM reasoning length varies by model, while context compression and reasoning failures are critical research areas.
• Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations — LLM robustness to CoT perturbations varies significantly by type and model scale, challenging assumptions that size alone ensures reliability.
• Less Is More: Cognitive Load and the Single-Prompt Ceiling in LLM Mathematical Reasoning — LLM mathematical reasoning accuracy plateaus despite extensive prompt engineering, limited by undecidability and prompt complexity.
• More Thinking, More Bias: Length-Driven Position Bias in Reasoning Models — Longer reasoning trajectories in LLMs increase position bias, a critical factor for evaluation reliability.
• Generalization in LLM Problem Solving: The Case of the Shortest Path — LLMs generalize spatially but fail at length scaling in shortest-path problems due to recursive instability.
• Studying the properties of large language models: an interview with Maxime Meyer — LLM performance on long inputs can be predicted and improved using specific formulas.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.