RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications

2026-07-07 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, medium

Summary

RuBench 1.0 is a new agentic coding benchmark designed to evaluate product-grade coding agents on real maintenance tasks specified natively in Russian, addressing a gap in existing English-centric benchmarks. It comprises 25 tasks derived from recent fix commits across five live open-source repositories: aiohttp, aiogram, Laravel, NestJS, and Fastify, covering Python, PHP, TypeScript, and JavaScript. Each task is authored from scratch in the style of a customer request, ensuring no training data contamination as all commits postdate evaluated models' cutoffs. The benchmark evaluates deployed product configurations, including Claude Code with Opus 4.8, Sonnet 5, and Haiku 4.5, and Codex CLI with GPT-5.5. The top-performing configuration achieved a 78.7% resolution rate. Auditing revealed that 20% of tasks (5 of 25) for one configuration were silently re-routed to Opus 4.8, highlighting that the product, not just the underlying model, is the true unit of measurement. Task statements, metadata, trajectories, and diffs are released, while grading oracles are withheld.

Key takeaway

For AI Engineers evaluating or deploying coding agents in multilingual environments, you should recognize that benchmarks like RuBench 1.0 highlight the critical need for native-language task specifications. Your evaluation strategy must extend beyond isolated LLM performance to scrutinize full product configurations, as silent model substitutions can significantly alter real-world outcomes. Prioritize testing agents with customer-style requests in their target languages to ensure robust, production-ready performance.

Key insights

Agentic coding benchmarks must use native-language, customer-style tasks to accurately measure deployed product performance, not just model capabilities.

Principles

• Real-world agent evaluation requires native language task specifications.
• Deployed product configurations are the actual units of measurement.
• Contamination control is vital for robust agentic coding benchmarks.

Method

Mine fix commits from live open-source repositories, author tasks natively as customer requests, and use withheld upstream regression tests for judging deployed CLI agent + model configurations.

In practice

• Develop agents to handle non-English, customer-style requests.
• Implement trajectory auditing to detect model fallbacks.
• Evaluate full agentic product stacks, not just LLMs.

Topics

• RuBench
• Agentic Coding
• Multilingual LLMs
• Software Engineering
• Benchmark Design
• Model Evaluation

Code references

• amazon-science/SWE-PolyBench
• SparksofAGI/CodeSpecBench

Best for: Research Scientist, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering — Current coding benchmarks fail to accurately evaluate agentic software engineering due to conflation, single-reference grading, and lack of component-level signal.
• 🧠 Sam Altman says OpenAI’s models are “beginning to find critical vulnerabilities” — AI models are advancing rapidly across security, coding, and voice synthesis, demanding new approaches to safety and efficiency.
• Claude Code’s 512,000-Line Leak Reveals Why AI Wrappers Are the Real Product — The recent 512,000-line source code leak of Claude Code unveiled a sophisticated, production-grade agentic system, revealing that advanced "AI wrappers" are the true product.
• MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights — Multilingual LLM evaluation requires fine-grained, agentic diagnosis to transform scores into actionable insights, addressing cultural and deployment realities.
• A Language For Agents — New programming languages designed for AI agents will emerge as coding costs decrease and agentic engineering grows.
• Code Reasoning for Software Engineering Tasks: A Survey and A Call to Action — LLM code reasoning improves through structured CoT, execution feedback, and inference scaling, culminating in agentic systems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.