Databricks Benchmarked Coding Agents on Its Own Codebase. The Results Should Change How You Buy

2026-07-11 · Source: Towards AI - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, medium

Summary

Databricks conducted an internal benchmark of coding agents on a sample of its polyglot codebase, which includes Scala, Go, Rust, Java, TypeScript, Protobuf, and Jsonnet. This study revealed four key findings that challenge conventional model selection. First, multiple models, including open-source options like GLM-5.2, now offer competitive top-tier performance on real engineering tasks, moving beyond Python-centric benchmarks like SWE-bench. Second, GLM-5.2 demonstrated strong performance even on Databricks' diverse codebase, validating its capabilities outside of standard benchmarks. Third, the minimal Pi harness achieved the same success rate as vendor-specific harnesses (e.g., Claude Code, Codex running Opus and GPT-5.5) at half the cost, primarily by using smaller LLM inputs. Finally, the "Price Reversal Phenomenon" was observed, where cheaper per-token models like Sonnet 5 resulted in higher total costs and lower quality per task compared to Opus 4.8 due to increased token usage.

Key takeaway

For AI Engineers or Directors of AI/ML evaluating coding agents, your model selection process should prioritize internal, task-specific benchmarks over public leaderboards. You must measure cost-per-completed-task on your own codebase, as cheaper per-token rates often lead to higher overall expenses and lower quality. Additionally, optimize your agent harnesses for minimal input, as this can halve costs while maintaining success rates. This approach ensures you select the most effective and economical solutions for your specific development environment.

Key insights

Internal, polyglot codebase benchmarks reveal true coding agent performance and cost-effectiveness beyond public metrics.

Principles

• Open-source models now compete with closed-source at the top tier.
• Minimal agent harnesses significantly reduce cost without sacrificing success.
• Per-token pricing does not predict per-task cost for agentic workloads.

Method

Databricks built an internal benchmark by sampling its own polyglot codebase, running candidate coding agents, and measuring success rate and cost-per-completed-task.

In practice

• Audit agent harness input footprint to identify cost savings.
• Measure cost-per-completed-task on your specific codebase.
• Consider open-source models like GLM-5.2 for diverse codebases.

Topics

• Coding Agents
• LLM Benchmarking
• Open-Source LLMs
• GLM-5.2
• Agent Harnesses
• Cost Optimization
• Price Reversal Phenomenon

Best for: CTO, AI Architect, VP of Engineering/Data, AI Engineer, Machine Learning Engineer, Director of AI/ML

Related on AIssential

• Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase — Benchmarking coding agents on real-world codebases reveals optimal performance requires diverse models and efficient harnesses, not just low token costs.
• Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering — Current coding benchmarks fail to accurately evaluate agentic software engineering due to conflation, single-reference grading, and lack of component-level signal.
• TensorBench: Benchmarking Coding Agents on a Compiler-Based Tensor Framework — Repository-level coding benchmarks require difficult tasks with robust, behavior-preserving evaluation to separate capability from noise.
• Databricks makes Chinese open-source model GLM 5.2 its default coding engine after it matched Opus at lower cost — Open-source models like GLM 5.2 can match top proprietary LLMs in coding performance at significantly lower costs.
• [AINews] FrontierCode: Benchmarking for Code Quality over Slop — FrontierCode redefines coding AI benchmarks by prioritizing mergeable, high-quality code over simple functional correctness.
• [AINews] Codex usage up >10x in 6 months to 7M users, +1M in the past ~day; did Codex overtake Claude Code?? — Rapid AI tool adoption and agentic system evolution underscore a shift towards outcome-driven, efficient, and secure AI deployments.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards AI - Medium.