An Exploratory Study on LLM-Generated Code and Comments in Code Repositories

2026-07-02 · Source: Artificial Intelligence · Field: Technology & Digital — Software Development & Engineering, Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

An exploratory study on LLM-generated code and comments in code repositories analyzed active company- and community-maintained repositories from 2021 to 2025. The research, using various detection tools, found that code likely generated by LLMs decreased over time and frequently appeared in test cases, while LLM-generated comments remained relatively stable. Company-maintained repositories showed a higher percentage of LLM-generated code and comments compared to community-maintained ones. Furthermore, LLM-generated code exhibited substantial intra-repository code clones, and comments often lacked grammatical correctness. Critically, only a small percentage of human-labeled bugs were associated with LLM-generated code.

Key takeaway

For software engineers integrating LLM-generated code, you should prioritize its use in test case generation, where its presence is decreasing but still frequent. While LLM-generated code shows low bug association, be vigilant for substantial intra-repository code clones. Additionally, carefully review LLM-generated comments for grammatical correctness, as their quality remains a concern.

Key insights

LLM-generated code decreased over time and appeared in test cases, while comments remained stable, with few associated bugs.

Principles

• LLM-generated code shows substantial intra-repository code clones.
• Company repositories use more LLM-generated content than community ones.
• LLM-generated comments often lack grammatical correctness.

Method

Conducted extensive experiments on active company- and community-maintained repositories from 2021 to 2025 using various LLM-generated code/comment detection tools.

In practice

• Focus LLM code generation on test cases.
• Review LLM-generated comments for grammatical accuracy.
• Monitor intra-repository code clones from LLM outputs.

Topics

• LLM Code Generation
• Code Repositories
• Software Engineering
• Code Quality
• AI in Development
• Code Comments

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Software Engineer

Related on AIssential

• The TechBeat: Can LLMs Generate Quality Code? A 40,000-Line Experiment (1/6/2026) — LLMs generate sloppy code over time, requiring formal analysis and multi-model reviews for quality assurance.
• How Software Engineering Students Use LLMs to Write Research Papers: An Experience Report — Students use LLMs across academic writing, balancing utility for clarity and organization with critical verification of AI-generated content.
• Automated Root-Cause Subclassification and No-Code Fix Generation for Invalid Bug Reports — Automating invalid bug report subclassification and no-code fix generation significantly reduces software maintenance overhead.
• CodeChat-Eval: Evaluating Large Language Models in Multi-Turn Code Refinement Dialogues — LLMs significantly degrade in functional correctness during multi-turn code refinement, especially with semantic or additive changes.
• LLMs and new programming languages: Complementary or Conflicting? — LLMs rapidly adapt to and support new programming languages, accelerating their adoption and reducing learning friction.
• Code Is More Than Text: Uncertainty Estimation for Code Generation — Code generation uncertainty estimation requires specialized methods accounting for code's unique properties, outperforming natural language-derived baselines.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.