Your coding agent doesn't always follow your rules — Talha Sheikh, Checkout.com

2026-07-08 · Source: AI Engineer · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, medium

Summary

AI coding agents, despite their advanced capabilities, often produce outputs requiring manual verification, leading to human developers acting as an "enforcement layer." Talha Sheikh of Checkout.com developed "Vector V1," a system that uses Claude hooks to deterministically check agent outputs and provide feedback for retries. While some believe future models like Anthropic's Project Claude Swing or Project Methuselah will eliminate the need for such enforcement, Sheikh argues that increased model capability does not equate to reliability. He posits that investing in robust verification "guardrails" or "harnesses" allows for the effective use of smaller, more cost-efficient models like Haiku or open-source alternatives, potentially reducing expenses compared to frontier models like Opus. This concept has evolved into a "language-agnostic contract" for universal verification, a pattern echoed by industry practices such as Anthropic's "executed advisor pattern" and OpenAI's "harness engineering." The core insight is a shift in value from code creation to verification design.

Key takeaway

For AI Engineers deploying coding agents, recognize that even advanced models require explicit verification. Instead of solely focusing on model capability, prioritize building robust, deterministic verification harnesses. This approach allows you to confidently use smaller, more cost-effective models while ensuring reliable outputs, shifting your investment from expensive frontier models to comprehensive testing frameworks. Actively design language-agnostic contracts for agent output validation to enhance trust and reduce operational costs.

Key insights

Model capability does not guarantee reliability; robust verification harnesses are essential for trustworthy AI agent outputs.

Principles

• Verification is distinct from instruction.
• Guardrails enable smaller, cheaper models.
• Value shifts from code to verification design.

Method

The author built "Vector V1" using Claude hooks to automatically check agent outputs against a config file of test cases, providing feedback for retries until tests pass.

In practice

• Implement deterministic checks for agent outputs.
• Design language-agnostic verification contracts.
• Prioritize building robust verification harnesses.

Topics

• AI Agents
• Code Generation
• Verification Systems
• LLM Reliability
• Testing Harnesses
• Cost Optimization

Best for: AI Architect, CTO, VP of Engineering/Data, AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Harness Engineering: The Missing Architectural Layer Between Powerful Models and Reliable AI Agents — The system around an AI model, not just the model itself, determines agent reliability and capability.
• I Gave Five AI Coding Agents a way to Fact-Check the Docs They Were handed. They Refused to Use it. — AI coding agents prioritize provided documentation over independent verification, even when it's wrong.
• LAI #133: The Most Anticipated Model of the Year and Most of You Skipped It — Recent AI advancements focus on agent memory, disciplined development, robust governance, and enhanced model capabilities.
• Your AI Agent Will Lie to You. Your Tests Won't. — AI agents optimize for reward, not correctness; robust, un-gameable oracles are essential for trustworthy code generation.
• From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agents — Shifting LLM agent behavior from prompts to auditable, code-owned contracts ensures enterprise reliability.
• Buffett Had Moody’s Manuals. I Built a Desk of AI Analysts. — Building trustworthy multi-agent AI systems requires rigorous engineering, particularly robust validation, retry logic, and honest evaluation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI Engineer.