Buffett Had Moody’s Manuals. I Built a Desk of AI Analysts.

2026-07-18 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, extended

Summary

The author developed a four-agent AI equity research desk using Claude Code, comprising a Screener (Haiku), Reader (Sonnet), Critic (Opus), and Synthesizer (Sonnet), orchestrated by plain Python. This system processes public SEC filings, generating cited research memos with machine-verified citations. A head-to-head comparison against a single-agent control across twelve screening queries, evaluated by a blind LLM judge, revealed the multi-agent desk guarantees 100% citation accuracy and provides a genuine adversarial bear case approximately two-thirds of the time. However, it costs \$1.25 per memo, double the single-agent's \$0.62, answers fewer screens (9 of 12 versus 11), and scores marginally lower on judge-rated quality. Critical engineering lessons included the necessity of typed handoff contracts with retry loops, which rescued 59% of initial 43% failures, and significant cost savings from caching, reducing costs by nearly 50%.

Key takeaway

For AI Engineers developing multi-agent systems, prioritize robust engineering over complex orchestration frameworks. Implement strict typed contracts and validate-and-retry loops at every agent handoff to manage LLM fallibility and ensure output reliability. Critically, invest in rigorously testing your evaluation harnesses, as measurement bugs can obscure true system performance. Your cost efficiency will largely depend on effective caching strategies, so design for memory reuse from the outset.

Key insights

Building trustworthy multi-agent AI systems requires rigorous engineering, particularly robust validation, retry logic, and honest evaluation.

Principles

• Multi-agent system boundaries are the product.
• Cost is dominated by caching strategies.
• Evaluation harnesses need rigorous testing.

Method

Orchestrate agents with plain Python, not a manager AI, using typed contracts at every handoff. Implement a validate-and-retry loop for agent outputs, and machine-verify all citations against source text.

In practice

• Implement typed contracts for agent handoffs.
• Prioritize caching to reduce LLM inference costs.
• Use behaviorally anchored rating scales for LLM judges.

Topics

• Multi-agent Systems
• LLM Evaluation
• Agent Orchestration
• Cost Optimization
• Citation Verification
• Pydantic Models

Code references

• briandonelan/multi-agent-equity-research-desk

Best for: AI Engineer, MLOps Engineer, Director of AI/ML

Related on AIssential

• Building an Agent Is Cheap Now. Proving It Works Is the Skill. — Defensible AI agent evaluation requires multi-dimensional grading, combining deterministic rules with a constrained LLM judge for nuanced, evidence-backed verdicts.
• [2026.05 Week 1] Five Trending Repos of the Week — Robust AI agent systems require sophisticated error handling, persistent memory, and secure, efficient execution environments.
• 🥇Top AI Papers of the Week — LLM internal states, external environments, and architectural choices profoundly impact performance, safety, and cost.
• Understanding AI Agents: What Actually Works When Building AI Products — Building robust AI products requires a structured approach addressing LLM limitations through wrapping layers, rigorous evaluation, and continuous operational feedback.
• not much happened today — AI development is rapidly advancing towards full-stack agentic systems, efficient inference, and adaptive learning, challenging traditional benchmarks.
• LAI #134: Your First LLM App on AWS for Under a Dollar — The newsletter highlights practical AI deployment, robust agent evaluation, and efficient LLM application development.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.