Output Format x Model Identity: Interaction Effects in Single-Round Coding Agent Performance

2026-07-27 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, extended

Summary

A controlled experiment investigated the interaction between output format and model identity in single-round coding agent performance, involving 3 models (DeepSeek V4, Doubao 2.0 Pro, Qwen 3.7 Max) and 3 output formats (full file, JSON Patch, unified diff) across 4,013 runs. The study found a significant format × model interaction, demonstrating no universally optimal format. Doubao achieved 94% success with JSON Patch (Cohen’s h = 1.57, p<0.001), DeepSeek excelled with unified diff (66%, h = 0.63), and Qwen showed a 50% success rate with full file (h = 0.29, p<0.05). Notably, only the tqdm project yielded non-zero success rates, with dotenv, requests, and jsoup resulting in zero successes across 2,551 runs. Researchers identified format misuse, such as applying a one-line fix as a full-file replacement, as a distinct failure mechanism. The study proposes a model-specific output strategy and a tool-design principle to constrain format semantics.

Key takeaway

For Machine Learning Engineers deploying LLM-based coding agents, defaulting to full-file output is a critical oversight. Your choice of output format significantly impacts agent success, with specific models excelling in different modalities. You should empirically test formats like JSON Patch and unified diff for your chosen model, as this can yield substantial performance gains. Implement tool-design principles that constrain format scope to prevent issues like one-line fixes becoming full-file replacements.

Key insights

Output format is not a neutral detail; it significantly interacts with model identity, determining coding agent success.

Principles

• Treat output format as a model-specific hyperparameter.
• Constrain format semantics to the agent's localization step.
• No single output format is universally optimal across all models.

Method

Propose a model-specific output strategy and a tool-design principle that constrains format semantics to the agent's own localization step, preventing excessive scope.

In practice

• Test multiple output formats for new coding models.
• Implement guardrails against excessive format scope.
• Consider JSON Patch for Doubao 2.0 Pro.

Topics

• Coding Agents
• LLM Output Formats
• JSON Patch
• Unified Diff
• Model Evaluation
• Software Engineering AI

Code references

• NousResearch/hermes-agent

Best for: AI Engineer, Research Scientist, AI Architect, AI Scientist, Machine Learning Engineer, Director of AI/ML

Related on AIssential

• Decoupling Task-Solving and Output Formatting in LLM Generation — Decoupling LLM task-solving from output formatting via a separate probabilistic model improves performance and guarantees compliance.
• Structured Output Collapses Answer Diversity Across 44 Language Models — Requesting structured output like JSON significantly reduces language model answer diversity, making responses more homogeneous.
• Experiences with local models for coding — Local small models for agentic coding are not plug-and-play, requiring careful task selection and setup.
• Lost in Space: Finding the Right Tokens for Structured Output — LLM structured output performance improves significantly with conventional formats and leading whitespace due to subword token representation benefits.
• JetBrains Tested Caveman Mode for AI Agents. The Result Is Useful, But Not Magical. — For AI coding agents, precise prompt engineering reduces token waste more effectively than output compression techniques like "Caveman mode."
• Get Vision LLMs to Follow Your Rules: Prompt-Guided JSON Formatting — Prompt-guided JSON formatting enables LLMs to generate structured data with specific, user-defined output formats.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.