Better Models: Worse Tools

2026-07-04 · Source: Simon Willison's Weblog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, quick

Summary

On July 4, 2026, Armin reported a problem where newer Anthropic Claude models, specifically Opus 4.8 and Sonnet 5, exhibit worse performance with custom edit tools like those in Pi compared to their older siblings. These "state-of-the-art" models sometimes invent extra, non-schema-compliant fields within the "edits[]" array during tool calls, causing Pi to reject the command. This issue is theorized to stem from the models being specifically trained, likely via Reinforcement Learning, to use Claude Code's baked-in edit tools effectively. This specialized training inadvertently leads to incorrect usage of third-party tool schemas, contrasting with OpenAI's Codex, which uses an "apply_patch" mechanism and is also trained for its specific tool.

Key takeaway

For AI Engineers integrating LLMs into custom coding harnesses, you should anticipate and mitigate tool call schema mismatches, especially with newer, "state-of-the-art" models like Anthropic's Opus 4.8 or Sonnet 5. Your systems must include robust validation layers for incoming tool calls to prevent rejections due to invented fields. Consider implementing model-specific tool interfaces to ensure compatibility and optimize performance across diverse LLM backends.

Key insights

Newer SOTA LLMs, optimized for specific internal tools, can degrade performance on external custom tool schemas.

Principles

• LLM tool training can be domain-specific
• Newer models are not universally superior
• Strict schema adherence is critical for tool integration

In practice

• Implement model-specific tool interfaces
• Rigorously validate LLM tool call schemas
• Test SOTA models with custom tool sets

Topics

• LLM Tool Use
• Anthropic Claude
• Schema Validation
• Code Generation
• Custom Tools
• Model Performance

Best for: AI Architect, NLP Engineer, CTO, AI Engineer, Machine Learning Engineer, AI Scientist

Related on AIssential

• Tool Calling Is Not an API Call: What Engineers Keep Getting Wrong — Tool calling in LLM agents demands robust engineering for reliability, unlike simple API integrations.
• llm-anthropic 0.25.1 — llm-anthropic 0.25.1 integrates Claude Opus 4.8, adds fast mode, and optimizes max_tokens defaults for better model interaction.
• References Improve LLM Alignment in Non-Verifiable Domains — Reference-guided LLM evaluators significantly improve model alignment and self-improvement in non-verifiable domains.
• Turning Andrej Karpathy’s LLM Coding Thoughts into Claude.md — LLMs are transforming coding workflows, shifting developers from direct coding to guiding and reviewing AI agents.
• The Model You Love Is Probably Just the One You Use — Developer preference for LLMs is often driven by familiarity and integration ease, not objective superiority.
• 4 Layer "AI Harness" For LLMs (+54%). Really? — The "AI Harness" improves LLM agent performance by addressing interface mismatches and syntax issues, not by modifying the frozen model.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Simon Willison's Weblog.