Toward Instructions-as-Code: Understanding the Impact of Instruction Files on Agentic Pull Requests

2026-06-11 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

This study investigates the impact of developer-created instruction files on AI-agent performance in generating pull requests (Agentic-PRs). Analyzing 15,549 agentic PRs from 148 projects in the AIDev dataset, researchers compared project metrics before and after instruction file creation. The findings indicate that providing instructions does not consistently improve agent performance; 27.7% of projects saw a merge rate increase of at least 20%, while 26.35% experienced a decrease. Similar mixed results were observed for code churn and merge effort. Initial exploration suggests that projects with improved merge rates utilized substantially longer and more structured instruction files, highlighting the need to treat instruction development as a formal "Instructions-as-Code" activity.

Key takeaway

For AI Engineers developing or integrating agentic systems for code generation, your approach to instruction files significantly impacts agent performance. Simply providing instructions is insufficient; focus on creating substantially longer and well-structured instruction files, treating this as an "Instructions-as-Code" discipline. Regularly analyze agentic PR merge rates and effort metrics to refine your guidance, ensuring your agents contribute effectively rather than increasing review burden.

Key insights

Instruction files for AI agents do not guarantee improved pull request merge rates or reduced effort.

Principles

• Instruction quality impacts agent performance.
• Longer, structured instructions correlate with better outcomes.
• Treat instructions as a software engineering activity.

Method

Researchers analyzed 15,549 agentic PRs across 148 projects, comparing merge rate, code churn, and merge effort before and after instruction file creation.

In practice

• Structure agent instructions with sections.
• Invest in detailed, comprehensive instruction files.
• Monitor agent PR metrics post-instruction changes.

Topics

• AI Agents
• Agentic Pull Requests
• Instruction Files
• Software Engineering
• Code Generation
• Instructions-as-Code

Best for: Machine Learning Engineer, NLP Engineer, Research Scientist, AI Scientist, AI Engineer, Software Engineer

Related on AIssential

• Toward Instructions-as-Code: Understanding the Impact of Instruction Files on Agentic Pull Requests — Effective AI-agent instruction files are substantially longer and more deeply structured, not just present.
• 10 Lessons for Agentic Coding — Agentic coding is transforming software development, requiring new approaches and skills.
• How I deleted 95% of my agent skills and got better results — Nick Nisi, WorkOS — Agent performance improves by enforcing actions with code, guiding with specific "gotchas," and rigorously measuring outcomes, not by comprehensive instruction.
• Faster Code, Slower Understanding: A Survival Note from a Year of Agent-First Engineering — AI shifts the engineering bottleneck from code production to comprehension and maintenance, demanding new optimization strategies.
• 🤖 AI Agents Weekly: Evaluating AGENTS.md, Perplexity Computer, Nano Banana 2, Doc-to-LoRA, Hermes Agent, Mercury 2, and More — Repository context files like AGENTS.md decrease coding agent performance and increase inference costs.
• Code Is Not Cheap: How to Multiply Your AI’s Output With Software Fundamentals — AI-assisted coding, while prevalent, demands human engineering expertise to mitigate quality issues from "jagged" model outputs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.