Feedback-to-Rubrics: Can We Learn Expert Criteria from Inline Comments?

2026-05-28 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, quick

Summary

A novel problem setting is proposed for learning reusable natural-language rubrics from accumulated inline comments on artifacts like human-written or LLM-generated drafts. This method addresses the challenge of eliciting tacit, undocumented, and context-dependent criteria, such as expert preferences or organization-specific conventions, which are crucial for effective LLM-based writing and review support. The approach infers rubrics from these comments and iteratively refines them by identifying comment-wise mismatches between rubric-conditioned predictions and reference comments. Evaluated in both real-world review settings and controlled environments with reference rubrics, the results demonstrate that inline comments can be effectively distilled into reusable rubrics. These learned rubrics support comment prediction, enhance rubric understanding, and facilitate automatic artifact revision.

Key takeaway

For NLP Engineers developing LLM-powered writing or review systems, this method offers a direct path to operationalize tacit expert criteria. You can distill organizational conventions and expert preferences from existing inline comments, significantly enhancing your model's contextual relevance. Implement this approach to generate reusable rubrics, improving comment prediction, fostering rubric understanding, and enabling more effective automatic artifact revision within your applications.

Key insights

Inline comments can be distilled into reusable natural-language rubrics, enhancing LLM-driven writing and review support.

Principles

• Expert criteria are often tacit and hard to elicit.
• Rubrics can be inferred from inline comments.
• Iterative refinement improves rubric accuracy.

Method

Rubrics are inferred from inline comments, then iteratively refined by observing comment-wise mismatches between rubric-conditioned predictions and reference comments.

In practice

• Predict comments based on learned rubrics.
• Improve understanding of review criteria.
• Automate artifact revision processes.

Topics

• Large Language Models
• Natural Language Processing
• Rubric Learning
• Inline Comments
• Automated Review
• Artifact Revision

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents — Integrating rubrics and contextual grounding significantly enhances LLM-generated peer review quality and substantiveness.
• Building AdvancedIF: Evolving Instruction Following Beyond IFEval and “Avoid the Letter C” — Current instruction-following benchmarks often prioritize programmatic verifiability over real-world utility, hindering AI development.
• Rubric-Based Rewards for RL — Rubric-based rewards extend RL to subjective LLM tasks by providing structured, interpretable evaluation criteria.
• Portuguese Sentiment Analysis with Open-Source LLMs: Models, Prompts, and Efficient Deployment — Open-source LLMs offer competitive Portuguese sentiment analysis with efficient deployment, especially using CoT+FS prompting.
• Evaluate generative AI models with an Amazon Nova rubric-based LLM judge on Amazon SageMaker AI (Part 2) — Amazon Nova's rubric-based LLM judge dynamically generates task-specific evaluation criteria for generative AI models.
• MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selection — MIRA integrates dynamic rubric construction into data selection for LLM mid-training, optimizing for heterogeneous sources.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.