Commenting with Copilot: A Taxonomy and Multi-Year Analysis of Student Code-Generation Specifications

2026-07-12 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

An analysis of a four-year dataset of undergraduate programming submissions reveals how students use comments as specifications for AI code generation tools like Copilot. The study introduces a taxonomy across three dimensions: comment type, code expression level, and code construct, classifying student comments using automated methods. Findings indicate students predominantly write natural-language "What" comments, describing desired outcomes. However, they shift towards "How" comments when dealing with more procedural code constructs. The research also highlights that students prioritize verifying generated code rather than repeatedly rewriting their initial natural language specifications. This work provides insights into the evolving interaction between students and AI-assisted programming environments, emphasizing the shift from direct code writing to effective specification.

Key takeaway

For AI Scientists designing future code generation tools, understanding student specification patterns is crucial. You should prioritize developing models that effectively interpret natural-language "What" comments, as students primarily use these to describe desired outcomes. Additionally, consider features that support the iterative verification of generated code, aligning with how students currently refine solutions rather than extensively rewriting initial specifications. This insight can inform more intuitive and efficient AI-assisted programming environments.

Key insights

Students primarily use natural language "What" comments for AI code generation, shifting to "How" for procedural tasks.

Principles

• Student focus shifts from code writing to specification.
• Comment types adapt to code construct complexity.
• Verification of generated code outweighs comment rewriting.

Method

A taxonomy of comment type, code expression level, and code construct was applied to a four-year student dataset using automated classification, analyzing attempts and reflections.

In practice

• Design AI tools to better interpret "What" specifications.
• Develop educational strategies for effective commenting.
• Focus AI feedback on generated code verification.

Topics

• AI Code Generation
• Copilot
• Student Programming
• Code Comments
• Natural Language Specifications
• Programming Education
• Taxonomy

Best for: AI Scientist, Research Scientist

Related on AIssential

• Commenting with Copilot: A Taxonomy and Multi-Year Analysis of Student Code-Generation Specifications — Students primarily use "What" comments for AI code generation, shifting to "How" for procedural tasks, and prioritize verification over comment rewriting.
• The Influence of Code Comments on the Perceived Helpfulness of Stack Overflow Posts — Code comments, especially block comments, significantly enhance the perceived helpfulness of Stack Overflow code snippets.
• Writing Code with Github Copilot and ChatGPT Together — III — AI code agents excel at routine tasks but require human oversight for novel or complex problems.
• ProjAgent: Procedural Similarity Retrieval for Repository-Level Code Generation — Procedural similarity, not just lexical or semantic, enhances repository-level code generation by matching functional logic.
• Is Code Better Than Language for Algorithmic Reasoning — Reliable external execution of code, not merely code as an intermediate representation, significantly improves algorithmic reasoning in LMs.
• Faster Code, Slower Understanding: A Survival Note from a Year of Agent-First Engineering — AI shifts the engineering bottleneck from code production to comprehension and maintenance, demanding new optimization strategies.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.