SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks

2025-01-01 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Advanced, extended

Summary

SWE-MERA is a dynamic, continuously updated benchmark designed to evaluate Large Language Models (LLMs) on software engineering tasks, addressing critical limitations of static datasets like SWE-bench. Previous studies revealed severe data contamination in SWE-bench, with 32.67% of successful patches involving direct solution leakage and 31.08% passing due to inadequate test cases. SWE-MERA employs an automated pipeline to collect real-world GitHub issues, ensuring quality and minimizing contamination risks. It currently offers approximately 10,000 potential tasks, with 300 samples available, collected between September 2024 and June 2025. Evaluation using the Aider coding agent demonstrates strong discriminative power across a dozen recent LLMs, including Qwen, Devstral, DeepSeek-R1, and Llama-3.3-70B-Instruct, and the benchmark is accessible via Hugging Face with a public leaderboard.

Key takeaway

For AI Scientists or ML Engineers evaluating LLMs for software engineering, you should prioritize dynamic benchmarks like SWE-MERA over static datasets. Traditional benchmarks like SWE-bench suffer from significant data contamination and saturation, compromising evaluation reliability. SWE-MERA's continuous updates and automated quality validation provide a more accurate and fair assessment of model capabilities. Consider integrating this benchmark into your evaluation pipeline to ensure robust and relevant performance metrics for your agentic LLMs.

Key insights

Dynamic, continuously updated benchmarks are crucial for reliable LLM evaluation in software engineering, mitigating data contamination and saturation.

Principles

• Static benchmarks cause data leakage.
• Dynamic updates ensure real-world relevance.
• LLM-based filtering maintains task quality.

Method

SWE-MERA employs a seven-stage pipeline: repository selection, PR-issue mapping, metadata extraction, patch validation, build validation, end-to-end execution, and LLM-based quality evaluation using Qwen3-32B.

In practice

• Download SWE-MERA from Hugging Face.
• Run your agent on the dataset.
• Submit results via pull request.

Topics

• Large Language Models
• Software Engineering Agents
• Dynamic Benchmarking
• Data Contamination
• GitHub Issues
• Aider Agent

Code references

• MERA-Evaluation/repotest
• Aider-AI/aider
• Aider-AI/aider-swe-bench
• DataDog/guarddog
• fkie-cad/socbed

Best for: AI Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Reliable and Developer-Aligned Evaluation of Agents for Software Engineering — Current LLM agent evaluations are unreliable; a new methodology is needed for real-world software engineering contexts.
• SWE-bench February 2026 leaderboard update — Independent SWE-bench Verified results show Claude 4.5 Opus leading in coding problem resolution.
• Open-SWE-Traces: Advancing Dual-Mode Multilingual Distillation for Software Engineering Agents — Open-SWE-Traces provides a large, dual-mode multilingual dataset to train agentic LLMs for autonomous software engineering.
• Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities — Current LLMs, even with agentic support, significantly underperform on real-world complex data analysis tasks.
• Exercises in benchmarking and evals, part 7: DeepSWE, Senior SWE-Bench, napkin math, and winter tires — Flawed methodology, unrepresentative tasks, and subjective scoring undermine benchmark validity and generalizability.
• Why Open-Source & Chinese LLMs Lead Coding Benchmarks, But Struggle in the Real World? — Benchmark saturation inflates LLM coding scores, necessitating fresh evaluation methods like SWE-rebench for true generalization.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.