Commenting with Copilot: A Taxonomy and Multi-Year Analysis of Student Code-Generation Specifications

2026-07-14 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Software Development & Engineering, Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A study titled "Commenting with Copilot: A Taxonomy and Multi-Year Analysis of Student Code-Generation Specifications" investigates how undergraduate students utilize natural language comments to specify code generation tasks with AI tools. Analyzing a four-year dataset of programming submissions and student reflections, the research introduces a taxonomy across three dimensions: comment type, code expression level, and code construct. The findings, derived from automated classification, reveal that students predominantly employ "What" comments, describing desired outcomes. They transition to "How" comments for more procedural code constructs. Crucially, students focused more on verifying the AI-generated code's correctness using test-case feedback rather than iteratively refining their initial comments. This analysis sheds light on evolving student interaction patterns with AI-assisted programming environments.

Key takeaway

For AI Product Managers designing code generation tools, understand that students primarily specify "What" they want, not "How." Your tools should prioritize robust code verification features, as users focus more on validating generated output than on iteratively refining initial natural language comments. Consider providing structured prompting guidance to help users articulate "How" for procedural tasks, improving initial generation quality and reducing verification effort.

Key insights

Students primarily use "What" comments for AI code generation, shifting to "How" for procedural tasks, and prioritize verification over comment rewriting.

Principles

• AI code tools shift focus to specification.
• Comment types vary by code construct.
• Verification dominates comment refinement.

Method

The study analyzed a four-year dataset of student programming submissions and reflections, introducing a three-dimensional taxonomy for comment classification, followed by automated classification to examine variations across attempts.

In practice

• Design prompts for "What" vs. "How" comments.
• Emphasize verification skills in AI-assisted coding.
• Analyze comment evolution for pedagogical insights.

Topics

• AI Code Generation
• GitHub Copilot
• Student Programming
• Natural Language Specifications
• Code Comment Taxonomy
• Software Engineering Education

Best for: Research Scientist, AI Scientist, AI Product Manager

Related on AIssential

• Commenting with Copilot: A Taxonomy and Multi-Year Analysis of Student Code-Generation Specifications — Students primarily use natural language "What" comments for AI code generation, shifting to "How" for procedural tasks.
• Same Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code — Explicitly labeling code as LLM-generated increases review time without improving thoroughness, driven by perceived origin.
• “A model that produces code which compiles and passes the tests it was given is not the same as a model that produces correct, secure, maintainable, well-architected software” — AI excels at code generation via next-word prediction but struggles with ensuring code robustness.
• Writing Code with Github Copilot and ChatGPT Together — III — AI code agents excel at routine tasks but require human oversight for novel or complex problems.
• From Generic to Personalized: Exploring Persona-Aware Code Review Explanations — Developer preferences for code review explanations vary significantly by problem-solving style, experience, and role.
• Claude Opus 4.6 vs GPT-5.3 Codex: How I shipped 93,000 lines of code in 5 days — Pairing generative AI models with specialized review models optimizes complex software development workflows.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.