RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications

2026-07-07 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

RuBench 1.0 is a new repository-level agentic coding benchmark designed to evaluate product-grade coding agents on real-world maintenance tasks specified in native languages. It features 25 tasks derived from recent fix commits in five live open-source repositories (aiohttp, aiogram, Laravel, NestJS, Fastify) across Python, PHP, TypeScript, and JavaScript. Each task is natively authored in Russian, mimicking actual customer requests, and judged by upstream maintainer regression tests, which are withheld. All fix commits postdate the training data cutoffs of evaluated models, ensuring a strong contamination argument. The benchmark evaluates deployed product configurations, including Claude Code with Opus 4.8, Sonnet 5, and Haiku 4.5, and Codex CLI with GPT-5.5, reporting pass@1, cost, and token usage. The top configuration achieved 78.7% task resolution. A critical finding from auditing a fifth configuration revealed silent model substitution on 20% of tasks, where HTTP-protocol fixes were re-routed to Opus 4.8, demonstrating that the deployed product, not just the model, is the true unit of measurement. Task statements, metadata, full agent trajectories, and diffs are released.

Key takeaway

For AI Engineers developing or deploying coding agents, you must consider native language task specifications and product-level behavior, not solely model performance. Your evaluation metrics should account for real-world customer request styles and potential silent model substitutions within agent products. This ensures robust, production-ready agent development and accurate performance assessment.

Key insights

Real-world coding agent benchmarks require native language tasks and reveal product-level measurement challenges beyond just model performance.

Principles

• Native language task specifications are crucial for real-world agent evaluation.
• Deployed product configurations, not just models, are the true unit of measurement.
• Benchmark integrity requires tasks postdating model training data cutoffs.

Method

Mine recent fix commits from live open-source repositories. Author tasks natively in customer-request style. Use upstream maintainer's regression tests for judging. Evaluate deployed CLI agent + model + reasoning effort configurations.

In practice

• Evaluate coding agents with native language, customer-style requests.
• Monitor for silent model substitutions in deployed agent products.
• Use post-cutoff commits to ensure benchmark integrity.

Topics

• RuBench
• Agentic Coding
• Benchmarking
• Large Language Models
• Code Generation
• Open-Source
• Russian Language

Best for: Research Scientist, Machine Learning Engineer, NLP Engineer, AI Scientist, AI Engineer, MLOps Engineer

Related on AIssential

• Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering — Current coding benchmarks fail to accurately evaluate agentic software engineering due to conflation, single-reference grading, and lack of component-level signal.
• A Language For Agents — New programming languages designed for AI agents will emerge as coding costs decrease and agentic engineering grows.
• Claude Code’s 512,000-Line Leak Reveals Why AI Wrappers Are the Real Product — The recent 512,000-line source code leak of Claude Code unveiled a sophisticated, production-grade agentic system, revealing that advanced "AI wrappers" are the true product.
• TensorBench: Benchmarking Coding Agents on a Compiler-Based Tensor Framework — Repository-level coding benchmarks require difficult tasks with robust, behavior-preserving evaluation to separate capability from noise.
• Finally a good benchmark (DeepSWE) — DeepSWE offers a robust, real-world software engineering benchmark revealing significant performance and cost disparities among LLMs.
• RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions — Agent benchmarks must reflect real-world deployed usage to accurately measure capability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.