"Go Home Copilot, You're Drunk": Understanding Developer Responses to Agent-Generated Code Review Comments

2026-07-27 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Software Development & Engineering, Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

A large-scale empirical study analyzed 54,713 agent-generated code review comments from Copilot, Cursor, and Codex across 341 Python repositories on GitHub to understand developer responses. The study found significant variation in resolution rates, with Copilot comments being resolved most frequently at 72.9%, while Codex had the lowest rate. Core developers resolved the majority of agent-generated feedback, particularly for design and evolvability-related comments, whereas peripheral developers focused more on functional defect comments. Analysis of unresolved comments revealed ten discussion patterns, with "incorrect suggestions" and "intentional design decisions" being the most common reasons for non-resolution. The presence of an inline code suggestion was identified as the strongest predictor of comment resolution (OR=1.62), while longer comments were less likely to be acted upon (OR=0.93).

Key takeaway

For AI Engineers developing code review agents, prioritize generating inline code suggestions, as they are the strongest predictor of comment resolution. You should also focus on integrating deeper project context to avoid "incorrect suggestions" and "intentional design decision" rejections. Tailor feedback by providing concise, actionable fixes for peripheral developers and higher-level insights for core developers, enhancing overall usefulness and trust in your agent's output.

Key insights

Actionable, context-aware AI code review feedback, especially with inline suggestions, significantly increases developer adoption.

Principles

• AI agents need project-level context for accurate feedback.
• Tailor AI feedback to developer experience levels.
• Inline code suggestions are key for comment resolution.

Method

The study collected 54,713 agent-generated comments, classified them using Llama-3.1-70B, categorized developers by experience, and applied open card sorting and logistic regression to analyze resolution patterns and influencing characteristics.

In practice

• Integrate inline code suggestions in AI review tools.
• Prioritize brevity for functional defect comments.
• Adapt explanation styles to issue context (e.g., rules for evolvability).

Topics

• AI Code Review
• Developer Experience
• Code Review Agents
• GitHub Repositories
• Large Language Models
• Software Quality Assurance

Code references

• mlflow/mlflow
• Azure-Samples/azure-search-openai-demo
• BerriAI/litellm
• celery/kombu
• comet-ml/opik

Best for: Machine Learning Engineer, Research Scientist, AI Product Manager, AI Scientist, Software Engineer, AI Engineer

Related on AIssential

• Commenting with Copilot: A Taxonomy and Multi-Year Analysis of Student Code-Generation Specifications — Students primarily use "What" comments for AI code generation, shifting to "How" for procedural tasks, and prioritize verification over comment rewriting.
• The TechBeat: Let the Agent Complain: A Feedback Loop for Better AI Tools (7/19/2026) — Agent feedback, beyond logs, reveals why AI tools fail, uncovering missing tools, context gaps, and product issues.
• Code Review is a Conversation: Toward Conversational AI Review Assistants — Code review is a conversation, not a one-shot commenting task, requiring AI assistants to be interactive partners.
• My cheatsheet for a clean context — Effective coding agent use requires mindful context management, precise prompting, and systematic debugging.
• The Engineering Manager’s Guide to AI Code Reviews — AI code generation creates a review bottleneck, demanding new processes focused on architectural integrity and testing.
• Code Is Not Cheap: How to Multiply Your AI’s Output With Software Fundamentals — AI-assisted coding, while prevalent, demands human engineering expertise to mitigate quality issues from "jagged" model outputs.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.