Toward Instructions-as-Code: Understanding the Impact of Instruction Files on Agentic Pull Requests

2025-12-25 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, long

Summary

A study analyzing 15,549 agentic pull requests (PRs) from 148 projects in the AIDev dataset investigated the impact of developer-created instruction files on AI-agent performance. The research found that while instruction files guide agents, they do not guarantee improved efficiency. Specifically, 27.7% of projects saw a merge rate increase of at least 20%, but 26.35% experienced a decrease. Projects with increased merge rates had significantly longer instruction files (median 976 words vs. 569 words) and more structured content, particularly with H3 headers. The study also observed mixed effects on task complexity and merge effort, with some projects showing increased complexity and others increased time/discussion for merging.

Key takeaway

For software engineers integrating AI agents like Copilot into development workflows, simply providing instruction files is insufficient. You should invest in creating comprehensive, well-structured instructions, similar to developing code. Projects with higher merge rates for agentic PRs utilized significantly longer files (median 976 words) with more detailed H3 sections. Treat instruction file creation as a critical software engineering task to maximize agent efficiency and reduce merge friction.

Key insights

Effective AI-agent instruction files are substantially longer and more deeply structured, not just present.

Principles

• Instruction files alone do not guarantee AI-agent performance improvement.
• Longer, well-structured instruction files correlate with higher agentic PR merge rates.
• Treating instruction file development as a software engineering activity is crucial.

Method

The study compared agentic PR metrics (merge rate, complexity, effort) before and after instruction file creation across 148 projects, using Mann-Whitney U tests and Cliff's delta for statistical significance.

In practice

• Prioritize detailed, multi-section instruction files for AI agents.
• Monitor agentic PR metrics after implementing instruction files.
• Structure instructions with H1, H2, and especially H3 headers for clarity.

Topics

• Agentic AI
• Pull Requests
• Instruction Files
• Software Engineering
• AI Agents
• Code Generation

Code references

• boxwise/boxtribute

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Software Engineer

Related on AIssential

• Writing a Good AGENTS.md — Effective `AGENTS.md` files are concise, human-written, and focus on essential, non-redundant information.
• Code Is Not Cheap: How to Multiply Your AI’s Output With Software Fundamentals — AI-assisted coding, while prevalent, demands human engineering expertise to mitigate quality issues from "jagged" model outputs.
• Article: From Prompts to Production: A Playbook for Agentic Development — Agentic AI development requires a specialized lifecycle and testing to manage nondeterministic behavior and ensure production readiness.
• GPT 5.4 is a big step for Codex — GPT 5.4 marks a practical leap for AI agents, excelling in reliability, speed, cost, and precise instruction following.
• Vibe coding is old now — AI agents are evolving rapidly, enabling non-coders to build software and raising critical security and usability questions.
• 10 GitHub Repositories To Master Claude Code — Mastering Claude Code requires understanding its ecosystem beyond basic prompting, leveraging structured agents and workflows.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.