The Model May Propose. The Governed System Must Dispose.

2026-07-19 · Source: AI on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Cybersecurity & Data Privacy · Depth: Advanced, long

Summary

The article argues against treating Large Language Models (LLMs) as sovereign controllers in agentic systems, proposing a "governed system" architecture. It highlights the dangers of conflating conversational fluency with operational authority, citing issues like misinterpreting tool outputs, long-horizon reliability problems (e.g., 0.98⁵⁰, or approximately 36.4 percent success over 50 steps), and security vulnerabilities such as prompt injection. The proposed "bounded authority" architecture places LLMs as capable instruments within a governed environment, emphasizing explicit state, task graphs, typed tool contracts, least authority, independent verification, provenance, reversibility, and visible authorization. This approach, termed the "Coherence Box," aims to maximize LLM usefulness in consequential work by ensuring accountability and control.

Key takeaway

For AI Architects designing agentic systems, you must prioritize a "governed system" over model-centric control. Conflating LLM fluency with operational authority introduces significant risks, including unreliable long-horizon operations and security vulnerabilities like prompt injection. Implement a bounded-authority architecture with explicit state, task graphs, and independent verification to ensure accountability and enable safe deployment of AI in consequential domains. This approach preserves LLM value by making capabilities legible and permissioned.

Key insights

LLMs should propose actions, but a governed system must dispose (decide, authorize, execute) to ensure reliability and safety.

Principles

• Conversational fluency does not imply operational authority.
• High local accuracy does not guarantee long-horizon reliability.
• Separate generative capability from operational permission.

Method

The "bounded authority" architecture involves a governed control plane with explicit state, task graphs, typed tool contracts, least authority, independent verification, provenance, and reversibility.

In practice

• Implement explicit state management for agent workflows.
• Decompose goals into inspectable task graphs.
• Distinguish model proposals from verified actions.

Topics

• AI Agent Architecture
• Large Language Models
• Operational Reliability
• Bounded Authority
• Prompt Injection
• System Governance

Best for: CTO, VP of Engineering/Data, AI Product Manager, AI Architect, AI Engineer, Director of AI/ML

Related on AIssential

• Your LLM Is Smart… So Why Can’t It Do Anything? — LLMs can reason about actions but require AI agents to actually execute them in real-world systems.
• Reliable AI Agents Need a Smaller Decision Surface — Reliable AI agents limit model autonomy to ambiguous decisions, delegating deterministic tasks to conventional software.
• The Model Called a Tool I Never Gave It — When LLM tools run on uncontrolled client devices, the architecture shifts from prompt engineering to trust and reliability.
• Most AI Agents Fail Because They Are Built Like Chatbots — AI agents need robust, external state management, not chat history, for reliable enterprise workflows.
• Podcast: [Video Podcast] Agentic Systems Without Chaos: Early Operating Models for Autonomous Agents — Agentic systems introduce a non-deterministic architectural domain requiring new operating models for control, reliability, and security.
• Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents — LLM agent failures compound nonlinearly with task length, despite progress in specific sub-tasks.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI on Medium.