#367 Don't Build on Jell-O: How to Make Agentic AI Reliable with Dan Klein, CTO at Scaled Cognition

2026-07-06 · Source: DataFramed · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Software Development & Engineering · Depth: Advanced, quick

Summary

Dan Klein, CTO and co-founder of Scaled Cognition and a UC Berkeley professor, addresses the critical issue of AI reliability lagging behind its rapidly advancing capabilities. He highlights how AI, despite fluent prose generation, can invent false information, leading to significant costs in high-stakes applications like banking and healthcare due to subtle, unnoticed errors. Klein's work at Scaled Cognition focuses on building trustworthy agentic AI, exemplified by APT (Agentic Pretrained Transformer), a frontier model designed from the ground up for reliable, policy-adherent operations. The discussion explores the prevalence of hallucinations, the limitations of human-in-the-loop and LLM-as-judge approaches, and the importance of integrating reliability directly into model architecture, emphasizing verifiable actions and test-driven agent development.

Key takeaway

For AI Engineers developing agentic systems, prioritizing reliability from the outset is crucial to prevent costly, subtle hallucinations. You should integrate verifiable actions directly into your model architecture and adopt test-driven agent development practices. This approach, championed by Scaled Cognition's APT, moves beyond reactive fixes like human-in-the-loop, ensuring your AI systems are inherently trustworthy and policy-adherent for high-stakes applications.

Key insights

AI reliability, particularly in agentic systems, requires architectural design for verifiable actions and test-driven development to counter pervasive hallucinations.

Principles

• Reliability must be designed into AI architecture.
• Hallucinations are often subtle and pervasive.
• Agentic AI needs verifiable actions.

Method

Build reliability into model architecture from the ground up, focusing on verifiable actions and employing test-driven agent development to mitigate hallucinations in agentic AI systems.

In practice

• Implement test-driven development for agents.
• Design AI systems for verifiable outputs.
• Evaluate LLM-as-judge limitations.

Topics

• AI Reliability
• Agentic AI
• Hallucinations
• Test-Driven Development
• Scaled Cognition
• APT Model

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Engineer, Machine Learning Engineer, AI Architect

Related on AIssential

• One Brain, Many Blind Spots — AI agents' confident hallucinations stem from training; multi-agent architectures provide essential verification for high-stakes applications.
• LAI #117: Why Reliable AI Systems Are Still So Hard to Build — AI agent development requires robust engineering for reliability, observability, and controlled autonomy to move beyond hype.
• Why Agentic AI Fails: Infinite Loops, Planning Errors, and More — Agentic AI system failures are predictable and stem from design flaws, not just model hallucinations.
• Google researchers introduce 'faithful uncertainty,' allowing LLMs to offer best guesses instead of hallucinations — Faithful uncertainty aligns LLM linguistic expression with internal confidence to mitigate confident errors and improve agentic AI reliability.
• The Three Pillars of Agentic AI Systems — A Developer’s Field Notes from Build with AI 2026 — Effective agentic AI systems require structured memory, controlled actions, and self-correction for real-world operation.
• Your Model Isn’t the Problem. Your Quant Is. — Model reliability in agentic workflows is critically determined by quantization and hardware, not just the base model.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by DataFramed.