Can Coding Agents Be General Agents?

2026-04-16 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Expert, long

Summary

This analysis investigates the generalizability of coding agents, such as Claude Code and GPT-5, to end-to-end business process automation beyond traditional software engineering tasks. It highlights significant improvements in coding capabilities, with SWE-Bench Verified scores rising from 49% to 78% and Terminal Bench scores from 43% to 61% by May 2025. The study identifies gaps in current evaluation benchmarks, noting that existing code-level evaluations (SWE-Bench, Terminal-Bench) lack business context, while domain-reasoning evaluations (BFCL, $\tau$-bench) lack complex code execution. A case study using a production-grade Odoo 19.0 Community Edition ERP system reveals that coding agents reliably complete simple tasks but exhibit characteristic failures on complex, multi-constraint scenarios. These failures include lazy code heuristics, business layer hallucinations, ignored policy constraints, and persistent overconfidence, suggesting that bridging domain logic and code execution is a critical bottleneck for generalizability.

Key takeaway

For AI Product Managers evaluating coding agents for business process automation, recognize that while these agents handle simple tasks effectively, their current limitations in complex domain logic, policy adherence, and prone-to-hallucination reasoning make them unreliable for critical, multi-constraint workflows. You should prioritize developing robust evaluation frameworks that measure both code execution and business correctness, and consider integrating domain-specific guardrails to mitigate risks of suboptimal outcomes and overconfidence in agent-driven processes.

Key insights

Coding agents excel at simple tasks but struggle with complex business logic and policy adherence, hindering generalizability.

Principles

• General agents require bidirectional business-code translation.
• Code-level success does not equate to business-level success.
• Asymmetric feedback leads to agent overconfidence.

Method

The study evaluates coding agents on an open-core ERP system (Odoo 19.0) using real-world business workflows with interdependent decisions and operational constraints, verifying outcomes against ground-truth database states.

In practice

• Test agents on full-stack business-code translation tasks.
• Implement robust business-level feedback mechanisms.
• Prioritize policy adherence in agent design.

Topics

• Coding Agents
• General AI Agents
• Business Process Automation
• Enterprise Resource Planning
• Odoo 19.0

Best for: Research Scientist, Machine Learning Engineer, AI Product Manager, AI Scientist, AI Engineer, MLOps Engineer

Related on AIssential

• How to Make a Coding Agent a General Purpose Agent - Harrison Chase — Coding agents, with robust harnesses and secure tool runtimes, form the foundation for scalable, general-purpose enterprise AI agents.
• Jun 16, 2026Economic ResearchAgentic coding and persistent returns to expertise — Domain expertise, not coding skill, is the primary driver of success and agent utility in agentic coding.
• The Open Agent Leaderboard — AI agent performance and cost depend on the full system, not just the underlying model.
• OpenClaw, Claude Code, and the Future of Software | Peter Yang on The a16z Show — AI agents are transforming work by automating tasks, enabling smaller teams, and shifting app interaction paradigms.
• QCon London AI Coding State of the Game: More Capable, More Expensive, More Dangerous Coding Agents — AI coding agents are more capable but introduce significant security risks and increased operational costs.
• How AI Coding Agents Actually Understand Your Codebase — AI coding agents use tools and an iterative loop to understand code, plan, and execute tasks, mimicking human engineers.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.