Hard Lessons Building Agents Since GPT-3.5

2026-04-22 · Source: Nicolas Bustamante · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Advanced, extended

Summary

The article, "Hard Lessons Building Agents Since GPT-3.5," details three years of experience developing AI agents at Fintool for professional investors, a domain where accuracy is paramount. The author highlights that traditional software engineering mindsets are obsolete, as code has become a commodity with a short lifespan due to rapid model advancements. Key lessons include treating English as the primary programming language, developing "model empathy" by "becoming the model," and embracing "eval-driven development" where robust evaluation sets are the only durable artifacts. The piece also stresses the importance of aggressive scaffolding deletion, comprehensive observability for non-deterministic systems, prioritizing quality over cost/latency, and hiring for "taste" rather than traditional credentials. The author's personal setup, involving multiple Claude Code terminals, exemplifies living at the technology's frontier.

Key takeaway

For AI Engineers and MLOps teams building agentic systems, recognize that traditional software development paradigms are obsolete. You must cultivate "model empathy" by deeply understanding how models interpret instructions and aggressively delete code scaffolding as new models emerge. Prioritize building robust, domain-specific evaluation sets as your primary durable asset, enabling rapid iteration and ensuring product quality. Your daily workflow should immerse you in agent interactions to develop the "taste" essential for shipping reliable, high-performing agents.

Key insights

Agent development demands a mindset shift from deterministic code preservation to embracing rapid iteration and model empathy.

Principles

• Code is a commodity; taste is valuable.
• Scaffolding has a short life expectancy.
• Evals are the durable spec.

Method

Implement "model-release drills" to chat with new models, run full eval suites, and identify scaffolding for deletion. Employ "eval-driven development" to create self-improving agent loops.

In practice

• Read prompts as if you were the model.
• Delete scaffolding when models advance.
• Live in an agent terminal daily.

Topics

• AI Agents
• LLM Development
• Agentic Systems
• Evaluation-Driven Development
• Prompt Engineering
• Observability
• Technical Debt

Best for: AI Engineer, MLOps Engineer, Director of AI/ML

Related on AIssential

• Steve Yegge Wants You to Stop Looking at Your Code — AI agents will transform coding into orchestration, shifting human roles to managing complex problems and leveraging taste.
• How Leading AI Teams Design Evals for Production Agents — Production AI agent failures stem from system design, context, and organizational issues, not just model quality.
• Why (Senior) Engineers Struggle to Build AI Agents — Philipp Schmid, Google DeepMind — Building AI agents demands a paradigm shift from deterministic software, embracing non-determinism, semantic understanding, and continuous evaluation.
• Vibe coding and agentic engineering are getting closer than I'd like — The increasing reliability of AI coding agents blurs the distinction between casual "vibe coding" and professional "agentic engineering."
• I Built a $0/Month Jarvis With an AI Pair Programmer, and Everything That Broke Along the Way — Real-world AI system development demands rigorous measurement, spec-driven workflows, and deep understanding of underlying components, often revealing validation metric limitations.
• Opus 4.6 and ChatGPT 5.3-Codex Are Here and the Labs Are at War — Leading AI labs are converging on coding agents as the foundation for general-purpose knowledge work agents.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Nicolas Bustamante.