Google updates Android Bench with new LLMs, but Gemini still lags behind

2026-07-08 · Source: AI - Ars Technica · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Intermediate, quick

Summary

Google has significantly updated its Android Bench, a benchmark designed to evaluate Large Language Model (LLM) performance in Android app development. The refresh introduces a new, easier-to-use Harbor framework and expands the leaderboard with eight new models, including Claude Fable 5, Claude Sonnet 5, and Qwen 3.7 Max. Notably, Claude Fable 5 now leads with 84.5% accuracy, while Google's own Gemini 3.1 Pro has fallen to fifth place, trailing models like GPT 5.4 and Claude Sonnet 5. The update also incorporates cost metrics, revealing that top performers like Fable 5 and GPT 5.5 incur over \$130 for the 100-problem benchmark, whereas Gemini 3.1 Pro costs \$87. Google encourages developers to contribute to Android Bench by submitting their own tests and feedback.

Key takeaway

For AI Engineers selecting LLMs for Android app development, your choice now involves a clearer trade-off between accuracy and cost. While Claude Fable 5 offers leading performance at 84.5% accuracy, be aware of its higher operational costs, exceeding \$130 per benchmark run. If cost-efficiency is paramount, consider models like Gemini 3.1 Pro, which, despite lower accuracy, runs for \$87. You should also explore contributing to Android Bench via the new Harbor framework to influence future evaluations.

Key insights

LLM performance for Android development varies significantly, with Google's Gemini models lagging behind competitors in accuracy but sometimes offering lower costs.

Principles

• LLM accuracy varies significantly across models for specific development tasks.
• Benchmarking LLMs should integrate both performance and operational cost metrics.
• Community contributions can enhance the relevance and scope of development benchmarks.

Method

The Harbor framework facilitates running, evaluating, sharing, and submitting custom Android development tasks to Android Bench, establishing new performance baselines.

In practice

• Consult Android Bench for current LLM performance and cost data in Android development.
• Utilize the Harbor framework to run and submit custom LLM evaluation tasks.
• Evaluate Claude Fable 5 for high-accuracy Android code generation, considering its cost implications.

Topics

• Android Bench
• LLM Benchmarking
• Code Generation
• Android Development
• Claude Fable 5
• Harbor Framework
• Operational Costs

Code references

• android-bench/android-bench

Best for: AI Architect, CTO, VP of Engineering/Data, AI Engineer, Machine Learning Engineer, Tech Journalist

Related on AIssential

• Gemini 3.5 Flash: more expensive, but Google plan to use it for everything — Gemini 3.5 Flash signals a new phase of higher LLM pricing and broad internal deployment across Google's ecosystem.
• I Switched From GPT-4 to Claude for My Production App — Here Is the Honest Comparison — Real-world production performance of LLMs often diverges significantly from standardized benchmark scores.
• Evaluation of LLMs for Mathematical Formalization in Lean — LLMs like Gemini 3.1 Pro and Claude Opus 4.7 show high effectiveness in generating formal Lean 4 proofs, with some models offering cost efficiency.
• Claude Fable 5 is here! — Claude Fable 5 offers unparalleled performance for complex, long-horizon tasks at a premium cost.
• How I Choose Between GPT, Claude, Gemini and Open Source Models for Every Task — Optimizing LLM usage requires a decision framework matching tasks to models for cost and quality.
• Gemini 3.1 Pro and the Downfall of Benchmarks: Welcome to the Vibe Era of AI — LLM performance is increasingly domain-specific, making overall "best model" claims misleading due to specialized post-training.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI - Ars Technica.