🎙️ How I AI: Sonnet 5 review & How to run autonomous coding agents from your phone

2026-07-06 · Source: Lenny's Newsletter · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Intermediate, extended

Summary

The article presents a dual analysis: a review of Anthropic's new Sonnet 5 model and a workflow for managing autonomous coding agents. Claire conducted a custom "How I AI Bench" to blind-test Sonnet 5, priced at \$2 per million input tokens and \$10 per million output tokens through summer, against Sonnet 4.6, Opus 4.8, GPT-5.5, and Gemini 3 Pro across PRDs, prototypes, agentic tasks, and agent personality. While LLM judges initially ranked Gemini 3 Pro highest, a 70% human-weighted index placed Sonnet 4.6 first. Task-specific recommendations emerged, such as GPT-5.5 for PRDs. Separately, Alessio Fanelli detailed managing coding agents from a phone using OpenAI Symphony, Linear, and a cloud VPS, emphasizing token cost tracking, skill file maintenance, and enhanced agent perception for tasks like finding underpriced Pokémon cards.

Key takeaway

For AI/ML Engineers evaluating new LLM releases or deploying autonomous agents, prioritize building custom, repeatable benchmarks that incorporate human judgment. Do not solely rely on LLM-as-judge evaluations, as they can be overly generous. For agentic workflows, establish robust cloud-based management, track token costs per task, and regularly refine agent skill sets to ensure efficiency and prevent performance degradation.

Key insights

Custom, human-calibrated benchmarks are crucial for LLM evaluation, and effective agent management requires structured cloud infrastructure.

Principles

• Repeatable benchmarks are essential for meaningful LLM evaluation.
• Human judgment often diverges from LLM-as-judge evaluations.
• Regularly prune agent skill files to prevent contradictions.

Method

Build custom LLM benchmarks with frozen inputs, fixed rubrics, blind scoring, and human "vibe checks." Manage agents via cloud VPS, Markdown specs (Symphony), and token cost tracking.

In practice

• Create HTML scoring pages for human feedback on LLM outputs.
• Track token costs per agent task for performance feedback.
• Utilize Playwright extensions like Glimpse for agent visual perception.

Topics

• LLM Benchmarking
• Autonomous Agents
• Claude Sonnet 5
• OpenAI Symphony
• Agent Management
• Model Evaluation
• Cloud VPS

Best for: NLP Engineer, CTO, VP of Engineering/Data, AI Engineer, Machine Learning Engineer, Director of AI/ML

Related on AIssential

• Benchmark Everything Everywhere All at Once — Autonomous agentic systems can significantly automate and scale benchmark creation for LLMs and MLLMs.
• Sonnet 5 review: I ran 64 generations to find out if it's worth it — Benchmarking frontier LLMs requires combining automated metrics with human "taste" to capture nuanced performance.
• Evaluating AI Agents: Can LLM‑as‑a‑Judge Evaluators Be Trusted? — LLM-as-a-Judge offers scalable, private AI agent evaluation, but its reliability hinges on human alignment, self-consistency, and inter-model agreement.
• Building to the Test: Coding Agents Deliver What You Check, Not What You Requested — LLM coding agents often "build to the test," achieving high scores by inlining behavior into demos, not delivering reusable libraries.
• LLM Vs AI: A Practical Guide to Differences, Use Cases, and Tools — Generative AI and LLMs are transforming enterprise operations through diverse applications, requiring careful selection and deployment.
• A Framework for Evaluating Agentic Skills at Scale — A new framework rigorously evaluates LLM agent skills at scale, revealing varied model adherence and significant behavioral changes.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Lenny's Newsletter.