Balancing Usefulness and Naturalness: An LLM-based Curation Pipeline for Code Review Comments

2026-07-13 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Expert, extended

Summary

Researchers propose two LLM-based curation pipelines, CuREV and CuREV+, to address the noise and inconsistency in existing code review datasets, which limit the effectiveness of LLMs in generating high-quality feedback. The initial dataset, comprising 176,613 samples, is first filtered to 170,718 relevant reviews. CuREV systematically reformulates all comments using Llama-3.1-70B to enhance clarity, conciseness, and civility, yielding average clarity scores of 8.96 and conciseness of 8.05, up from 6.89 and 7.71 respectively. CuREV+ selectively reformulates low-quality ("poor") comments, preserving 92,894 high-quality ("good") ones as in-context exemplars. This yields similar clarity (8.95) but superior conciseness (8.53) and greater stylistic diversity. Both curated datasets improve downstream tasks, with CuREV+ achieving the highest CodeBLEU (0.49) and Exact Match (463) for code refinement, compared to 0.36 and 408 for the original dataset.

Key takeaway

For Machine Learning Engineers developing automated code review systems, your LLM's performance is directly tied to training data quality. You should prioritize implementing a selective, exemplar-guided curation pipeline, such as CuREV+, to refine noisy datasets. This approach significantly improves comment generation (BLEU up to 11.05) and code refinement (CodeBLEU up to 0.49) by balancing clarity, conciseness, and stylistic diversity, leading to more accurate and human-like feedback.

Key insights

Curating LLM training data for code review comments by balancing quality and natural diversity significantly improves model performance.

Principles

• LLM training data quality directly impacts model utility.
• Curation must balance standardization with stylistic diversity.
• In-context exemplars enhance realism in LLM reformulations.

Method

Filter irrelevant comments, then classify remaining into high/low quality. Preserve high-quality comments and use them as in-context exemplars for LLM-based reformulation of low-quality comments.

In practice

• Implement LLM-as-a-Judge for dataset quality assessment.
• Filter out irrelevant comments using a relevance score threshold.
• Use in-context learning with high-quality examples for LLM text refinement.

Topics

• Code Review
• Large Language Models
• Dataset Curation
• Code Refinement
• Comment Generation
• Software Engineering

Code references

• AI4CodeReview/CuREV
• OussamaSghaier/CuREV
• OussamaSghaier/CuREV-plus

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Software Engineer

Related on AIssential

• Are LLMs safe? — LLM training data curation, especially via quality filters, embeds implicit biases that shape model behavior and capabilities.
• Evaluating Semantic and Quality-Aware Retrieval for Source Code Repositories — Semantic retrieval excels for general code search, while quality metadata enhances quality-specific queries, but can distort topical relevance.
• The Argonauts at SemEval 2026 Task 6: Large Language Models for Response Clarity Classification: Prompting, Fine-Tuning, and Data-Centric Approaches — Detecting equivocation in political responses benefits from data-centric LLM fine-tuning with class rebalancing.
• Don’t Let Claude Grade Its Own Homework — Cross-provider LLM review, with structured feedback and automated author response, scales code quality assurance.
• LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation — LLMs can generate synthetic data to validate NLP evaluation metrics, replacing costly human annotations.
• Towards Knowledge Alignment in Code LLMs: Contrastive Unlearning for Evolving APIs — CURE uses contrastive unlearning to both suppress deprecated APIs and promote valid replacements in Code LLMs.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.