Better Models: Worse Tools

2026-07-04 · Source: Armin Ronacher's Thoughts and Writings · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, long

Summary

Newer Anthropic models, specifically Opus 4.8 and Sonnet 5, exhibit a regression in tool calling, generating malformed arguments for Pi's edit tool by inventing extra fields in the "edits[]" array. This issue, observed since July 04, 2026, causes Pi to reject otherwise correct edit payloads. The problem is more prevalent in advanced models than their predecessors and is context-dependent, appearing in complex agentic histories. The author posits this deterioration stems from post-training within a "slop harness" like Claude Code, which tolerates and repairs malformed tool calls, inadvertently reinforcing non-schema-compliant outputs. This adaptation to a specific, forgiving tool ecology makes models less effective with alternative, stricter schemas. Enabling "strict" tool invocation on Anthropic's side appears to resolve the issue.

Key takeaway

For AI Engineers integrating Anthropic models with custom tools, recognize that newer models like Opus 4.8 may struggle with complex or nested schemas. You should enable "strict" tool invocation to enforce schema adherence, preventing malformed calls. Alternatively, simplify your tool definitions to align with flatter structures, potentially mirroring Claude Code's internal patterns. Ignoring this risk means your agentic workflows could experience frequent, context-dependent tool call failures.

Key insights

Newer, highly capable LLMs can degrade in tool schema adherence due to training on forgiving, undocumented "slop harnesses."

Principles

• Tool schemas are not neutral; models adapt to specific training ecologies.
• Forgiving RL environments can reinforce non-schema-compliant outputs.
• Grammar-constrained decoding provides stronger output guarantees.

In practice

• Enable Anthropic's "strict" tool invocation mode.
• Design tool schemas to align with common model training priors.
• Implement robust client-side validation and retry paths.

Topics

• Anthropic Models
• LLM Tool Calling
• Constrained Decoding
• Schema Validation
• Reinforcement Learning
• Claude Code

Code references

• earendil-works/pi

Best for: AI Architect, AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Anthropic killed Tool calling — Anthropic's tool calling updates enhance agent efficiency and accuracy for complex tasks by optimizing token use and improving tool interaction.
• Constraint Tax in Open-Weight LLMs: An Empirical Study of Tool Calling Suppression Under Structured Output Constraints — Joint JSON Schema and Tool Calling constraints suppress tool invocation in open-weight LLMs due to token mask conflicts.
• Anthropic’s Claude Opus 4.7 is finally here — Claude Opus 4.7 improves coding and reasoning, but new safety measures slightly reduce cybersecurity vulnerability reproduction.
• Tool Calling Is Not an API Call: What Engineers Keep Getting Wrong — Tool calling in LLM agents demands robust engineering for reliability, unlike simple API integrations.
• llm-anthropic 0.25.1 — llm-anthropic 0.25.1 integrates Claude Opus 4.8, adds fast mode, and optimizes max_tokens defaults for better model interaction.
• 😺 Anthropic shipped Opus 4.7. OpenAI countered. — New AI models from Anthropic and OpenAI offer enhanced capabilities but introduce complexities like increased token usage and account automation risks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Armin Ronacher's Thoughts and Writings.