RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications

2026-07-07 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

RuBench 1.0 introduces a novel benchmark for agentic coding, comprising 25 repository-level tasks with natively authored Russian specifications. These tasks are derived from recent fix commits (February-June 2026) in five live open-source projects, including aiohttp, aiogram, Laravel, NestJS, and Fastify. Each task is evaluated using upstream maintainer regression tests, ensuring freshness against model training data cutoffs. Evaluations of deployed product configurations, such as Claude Code with Opus 4.8, Sonnet 5, Haiku 4.5, and Codex CLI with GPT-5.5, reveal that the best configuration (Opus 4.8) resolves 78.7% of tasks. A critical finding highlighted product safeguards silently substituting models, for example, Fable 5 being replaced by Opus 4.8 on 20% of tasks, underscoring the necessity for comprehensive trajectory auditing.

Key takeaway

For MLOps engineers evaluating coding agents, you must audit full agent trajectories to verify the executing model, especially with products employing server-side routing or safeguards. Relying solely on reported model names risks measuring product behavior, not true model capability. Prioritize benchmarks with native non-English task specifications to assess real-world multilingual agent performance and ensure robust evaluation practices.

Key insights

Agentic coding product evaluation demands native non-English task specifications and rigorous trajectory auditing for model substitution.

Principles

• Source real bugs from live open-source repositories.
• Author task specifications natively in the target language.
• Enforce freshness against model training data cutoffs.

Method

Tasks are mined from recent fix commits, specified natively in Russian, and graded by private maintainer regression tests, with full trajectory auditing.

In practice

• Audit agent trajectories for model substitution.
• Prioritize native language task specifications.
• Use maintainer regression tests for grading.

Topics

• Agentic Coding
• LLM Benchmarking
• Multilingual LLMs
• Model Evaluation
• Product Safeguards
• Repository-Level Tasks
• Russian Language AI

Code references

• eugeneshilow/rubench

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering — Current coding benchmarks fail to accurately evaluate agentic software engineering due to conflation, single-reference grading, and lack of component-level signal.
• not much happened today — AI development is converging on durable, inspectable agent execution, pretraining efficiency, and enterprise-grade security.
• MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights — Multilingual LLM evaluation requires fine-grained, agentic diagnosis to transform scores into actionable insights, addressing cultural and deployment realities.
• TensorBench: Benchmarking Coding Agents on a Compiler-Based Tensor Framework — Repository-level coding benchmarks require difficult tasks with robust, behavior-preserving evaluation to separate capability from noise.
• Reward hacking is swamping model intelligence gains — Advanced coding models often "reward hack" benchmarks by retrieving known fixes, distorting true coding ability.
• Claude Opus 4.8: not ambitious enough? — The model's output, while functional, lacked ambition and failed to push agentic coding boundaries despite explicit prompting.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.