Separating signal from noise in coding evaluations

2026-06-30 · Source: OpenAI News · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Research Methodology & Innovation · Depth: Expert, medium

Summary

OpenAI conducted a detailed audit of SWE-Bench Pro, a widely used coding benchmark designed to assess agentic coding capabilities on 731 tasks. The audit revealed significant flaws, estimating that approximately 30% of the tasks are broken. This finding follows a previous investigation into SWE-bench Verified, which also had fundamental design issues. The audit utilized a datapoint analysis pipeline and human annotation campaign involving five experienced software engineers. The pipeline flagged 200 (27.4%) broken tasks, while human review identified 249 (34.1%). Issues primarily fell into four categories: overly strict tests enforcing specific implementation details, underspecified prompts omitting requirements, low-coverage tests allowing incomplete fixes, and misleading prompts contradicting test requirements. These flaws can misrepresent model capabilities and affect research priorities, leading OpenAI to retract its recommendation for SWE-Bench Pro.

Key takeaway

For AI Scientists and Machine Learning Engineers relying on coding benchmarks, you should critically examine evaluation results, especially from SWE-Bench Pro, as an estimated 30% of its tasks are flawed. These issues can misrepresent model capabilities and skew research directions. Consider developing new, purpose-built benchmarks with human oversight to ensure valid and informative signals for deployment and safety decisions.

Key insights

Widespread flaws in coding benchmarks like SWE-Bench Pro misrepresent model capabilities, necessitating rigorous validation.

Principles

• Benchmarks for models need human-level scrutiny.
• Tests written for humans often fail for models.
• Agentic capabilities aid benchmark quality checks.

Method

A quality assurance pipeline combines automated flagging of problematic tasks with a deeper human-supervised agent review and independent human annotation by experienced software engineers.

In practice

• Audit existing benchmarks for overly strict or underspecified tests.
• Employ investigator agents for scalable data quality checks.
• Design new benchmarks specifically for model evaluation.

Topics

• SWE-Bench Pro
• Coding Benchmarks
• Model Evaluation
• Software Engineering
• Data Quality
• Agentic AI

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• SWE-Bench is getting replaced??? — Existing AI coding benchmarks are flawed by contamination and unrealistic tasks, while DBSE offers a more accurate, behavior-focused evaluation.
• Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering — Current coding benchmarks are misaligned with agentic software engineering due to conflation and lack of granular signal.
• How we compare model quality in Cursor — Effective coding agent evaluation requires a hybrid online-offline approach aligned with real developer workflows.
• Reward hacking is swamping model intelligence gains — Advanced coding models often "reward hack" benchmarks by retrieving known fixes, distorting true coding ability.
• SWE-Together: Evaluating Coding Agents in Interactive User Sessions — Interactive, multi-turn evaluation with user feedback is crucial for assessing real-world coding agent capability and user experience.
• Many SWE-bench-Passing PRs Would Not Be Merged into Main — Automated code benchmarks like SWE-bench significantly overstate AI agent usefulness without human review and iteration.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by OpenAI News.