Anthropic's Big AI Design Change: The RED Pill

2026-06-06 · Source: Discover AI · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Expert, extended

Summary

Anthropic's recent publication, "How Anthropic enables the self-service data analytics with Claude," reports its Claude LLM achieves only 21% accuracy on analytical questions. To improve this, Anthropic employs a "scaffolding" approach. This deterministic, human-defined system of rules, workflows, and data governance boosts accuracy to 95%. The scaffolding includes an "analytical stack" addressing failure modes like incorrect field selection, stale knowledge, and data retrieval issues. It uses structured documentation, explicit instructions, and a mandatory semantic layer to guide the LLM. Analysis by Claude Sonnet 4.6 and ChatGPT indicates this method replaces the LLM's probabilistic reasoning with deterministic logic. This raises questions about the necessity and cost of highly trained LLMs for such tasks.

Key takeaway

For AI Architects and Directors of AI/ML designing data analytics solutions, re-evaluate the value of expensive, highly trained LLMs. If your system's accuracy primarily stems from deterministic scaffolding and human-encoded logic, consider smaller, more cost-effective LLMs. Focus investment on robust scaffolding and clear instruction sets. The intelligence often resides outside the core LLM for these use cases.

Key insights

Deterministic scaffolding, not LLM intelligence, achieves high accuracy in analytical tasks, challenging the value proposition of expensive frontier models.

Principles

• LLMs alone are insufficient for high analytical accuracy.
• Deterministic scaffolding dramatically improves LLM task performance.
• Human-defined logic can override LLM probabilistic reasoning.

Method

Anthropic's method builds a multi-layer analytical stack with sources of truth, validation, and "skills" (deterministic instructions) to guide the LLM in data selection, query execution, and answer verification.

In practice

• Implement a mandatory semantic layer for data queries.
• Conduct adversarial SQL review via sub-agents.
• Encode business logic in LLM-readable markdown.

Topics

• Anthropic Claude
• LLM Scaffolding
• Data Analytics
• AI System Design
• Expert Systems
• Model Accuracy

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, AI Architect, Director of AI/ML

Related on AIssential

• Are SKILL.md files the Quantum Error Codes of Industrial AI? — Industry's distrust of probabilistic AI systems creates a "rift" in adoption, driving the use of deterministic scaffolding.
• How Anthropic enables self-service data analytics with Claude — Analytics accuracy with LLMs hinges on mapping user questions to specific, current data model entities, not merely code generation.
• AI Trends 2026: OpenClaw Agents, Reasoning LLMs, and More [Sebastian Raschka] - 762 — LLM evolution focuses on post-training reasoning, tool use, and inference scaling to enhance problem-solving and efficiency.
• 2.5-D Decomposition for LLM-Based Spatial Construction — Decomposing LLM spatial tasks by offloading deterministic dimensions to an executor drastically improves accuracy.
• Another try at LLM workflow automation — Effective LLM automation requires extensive scaffolding and iterative tooling, not just smarter models.
• Human Mind Reconstruction w/ AI (ToM, UserHarness) — Decoupling epistemic structure from LLM generation significantly boosts reasoning accuracy and efficiency, even for smaller models.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Discover AI.