Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase

2026-07-08 · Source: Databricks · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Intermediate, medium

Summary

Databricks developed an internal benchmark to evaluate coding agents on real-world tasks against its multi-million line codebase, spanning languages like Python, Go, Typescript, and Scala. The analysis revealed that achieving frontier performance requires a mix of OpenAI, Anthropic, and open-source models. Notably, open models like GLM 5.2 can handle high-difficulty tasks, matching Opus 4.8's quality at a lower cost (\$1.28/task vs. Opus's \$1.94). The study also found that token price is a poor predictor of actual task costs, as more token-efficient larger models can be cheaper overall (e.g., Sonnet 5 was 1.7x cheaper per token but cost \$2.09/task compared to Opus's \$1.94). Furthermore, the choice of harness significantly impacts cost and quality, with simpler harnesses like Pi demonstrating better efficiency by managing context more effectively. Models clustered into three capability tiers, indicating that task complexity should guide model selection.

Key takeaway

For AI/ML Engineering Directors evaluating coding agent deployments, you should move beyond token cost as a primary metric and instead benchmark agents on your actual codebase. Implement a mixed-model strategy, leveraging open models like GLM 5.2 for common tasks and more expensive models for complex problems. Critically, experiment with different harnesses, as their context management significantly impacts overall cost and quality, allowing you to optimize efficiency and avoid vendor lock-in.

Key insights

Benchmarking coding agents on real-world codebases reveals optimal performance requires diverse models and efficient harnesses, not just low token costs.

Principles

• Optimal coding agent performance requires a mix of models.
• Token price is a poor indicator of true task cost.
• Harness choice dramatically impacts agent efficiency.

Method

Databricks' benchmark uses filtered, recent, human-written PRs from its codebase, converting them into well-specified prompts with associated high-quality tests. It evaluates agent output against these tests, sealing Git history to prevent solution recovery.

In practice

• Evaluate coding agents on your own codebase's PRs.
• Experiment with different harnesses for cost efficiency.
• Route tasks to models based on complexity tiers.

Topics

• Coding Agents
• LLM Benchmarking
• Databricks
• GLM 5.2
• AI Gateway
• Omnigent
• Code Generation

Best for: CTO, VP of Engineering/Data, AI Architect, AI Engineer, Machine Learning Engineer, Director of AI/ML

Related on AIssential

• Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering — Current coding benchmarks fail to accurately evaluate agentic software engineering due to conflation, single-reference grading, and lack of component-level signal.
• Open source llm (glm 4.7) matching closed models on coding benchmarks. Tested via api on real projects. — Open-source models are achieving competitive, specialized performance in domains like coding at significantly lower costs.
• Opus 4.6 and ChatGPT 5.3-Codex Are Here and the Labs Are at War — Leading AI labs are converging on coding agents as the foundation for general-purpose knowledge work agents.
• TensorBench: Benchmarking Coding Agents on a Compiler-Based Tensor Framework — Repository-level coding benchmarks require difficult tasks with robust, behavior-preserving evaluation to separate capability from noise.
• Benchmarking AI Agents on Kubernetes — AI coding agents struggle with system-wide impact analysis, often providing incomplete fixes despite effective code retrieval.
• GLM 5.1 Thinks Strategically, Data-Center Revolt Intensifies, When Helpful LLMs Turn Unhelpful, Humanoid Robots Get to Work — AI agents accelerate software development unevenly, with frontend seeing the most gains, while new models and robotics advance, and data center growth faces public backlash.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Databricks.