I benchmarked the NEW Sonnet 5. The results shocked me.

2026-06-30 · Source: How I AI · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, extended

Summary

Anthropic's new Claude Sonnet 5 model is positioned to offer Opus-level performance at Sonnet prices, featuring enhanced agentic tool use and capabilities for computer and knowledge work. Priced at \$2 per million input tokens and \$10 per million output tokens through the end of summer, its release prompted the development of the "Howi AI Bench." This benchmark, combining AI and human-graded evaluations, assesses models on tasks like PRD writing, bug solving, and design prototyping. Initial blind tests surprisingly placed Gemini 3 Pro and Sonnet 5 at the top, tied with GPT 5.5, while Opus 48 and Sonnet 46 scored lower. However, the human "vibe check" scores often diverged significantly from automated LLM judgments, highlighting the challenge of capturing qualitative "taste." A final weighted leaderboard, prioritizing human judgment, ultimately ranked Sonnet 46 and Gemini 3 Pro highest, with Sonnet 5 and Opus 48 at the bottom, alongside task-specific model recommendations.

Key takeaway

For AI/ML engineers evaluating new frontier models, you should integrate qualitative human "vibe checks" with automated benchmarks to get a comprehensive view of model performance. Relying solely on LLM-graded evaluations may obscure critical nuances like "taste" or specific functional issues, leading to suboptimal model selection for tasks like creative prototyping or agentic voice. Prioritize benchmarks that reflect real-world application and human preference.

Key insights

Effective LLM benchmarking requires balancing automated metrics with nuanced human qualitative judgment to capture true model utility.

Principles

• LLM self-grading tends towards mediocrity, lacking "spiky" evaluations.
• Human "taste" is critical for assessing subjective outputs like creative designs or voice.
• Some agentic tasks are too uniformly handled by top models to serve as strong differentiators.

Method

Develop custom benchmarks using LLMs to brainstorm tasks and design principles. Integrate both automated LLM scoring and human "vibe checks," then combine results using a weighted index to reflect desired emphasis.

In practice

• Use Claude Code to generate tailored evaluation benchmarks for new LLMs.
• Incorporate human qualitative reviews to capture subjective model performance.
• Consider GPT 5.5 for PRD writing and Sonnet 46 for prototyping or conversational agents.

Topics

• LLM Benchmarking
• Claude Sonnet 5
• Gemini 3 Pro
• GPT 5.5
• Model Evaluation
• Agentic AI
• Product Design

Best for: AI Engineer, Machine Learning Engineer, Director of AI/ML

Related on AIssential

• Sonnet 5 review: I ran 64 generations to find out if it's worth it — Benchmarking frontier LLMs requires combining automated metrics with human "taste" to capture nuanced performance.
• From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs — Formalized "vibe-testing" bridges the gap between LLM benchmark scores and real-world user utility by personalizing evaluation.
• [D] What is even the point of these LLM benchmarking papers? — LLM benchmarking papers often become obsolete quickly due to rapid model updates, but their datasets can still be valuable.
• Human Evaluation of Large Language Models: How Good is Hugging Face’s BLOOM? — Human evaluation is crucial for assessing large language models' real-world performance beyond flawed academic benchmarks.
• True Zero-shot MT — LLM evaluation faces a benchmark crisis, while specific models like Command R+ excel in RAG and multilingual tasks.
• AI Outperforms Humans in Personalized Image Aesthetics Assessment via LLM-Based Interviews and Semantic Feature Extraction — An integrated DL-LLM system accurately predicts individual image aesthetics by combining low-level and high-level feature extraction.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by How I AI.