DPO-F+: Aligning Code Repair Feedback with Developers' Preferences

2026-07-09 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

The DPO-f+ framework enhances Large Language Model (LLM) generated code repair feedback by aligning it with developer preferences, addressing challenges in human-AI teaming. This novel approach formalizes developer-profiled, domain-specific metrics for feedback alignment, automatically constructs pairwise preference datasets from code-repair tasks, and fine-tunes LLMs using Direct Preference Optimization (DPO) augmented with a lightweight margin signal. It also includes an automated feedback evaluation protocol. Empirically, DPO-f+ significantly outperforms both baseline models and standard DPO. On novice programming tasks, it boosts Pass@1 by 5.71 percentage points (pp) over the baseline and 3.30 pp over DPO. For the more complex SWE-bench Lite benchmark, it increases the issue-resolution rate by 1.67 pp over DPO and 4.67 pp over the baseline, while also achieving superior feedback alignment.

Key takeaway

For Machine Learning Engineers developing code repair LLMs, you should integrate preference-based alignment frameworks like DPO-f+ to improve both code accuracy and developer comprehension. By tailoring feedback to specific developer profiles and task contexts, your models can foster more effective human-AI collaboration, reducing clarification cycles and enhancing overall workflow efficiency in software development. Consider implementing a reward-augmented DPO approach to achieve superior alignment.

Key insights

DPO-f+ aligns LLM code repair feedback to developer profiles, improving comprehension and human-AI teaming.

Principles

• Feedback alignment requires profile-aware metrics.
• Preference margins enhance DPO training.
• Automated evaluation scales feedback quality.

Method

DPO-f+ constructs preference datasets using developer-profiled metrics, then fine-tunes LLMs with a reward-augmented DPO loss, and evaluates using an automated protocol.

In practice

• Tailor LLM feedback for novice programmers.
• Improve code review efficiency with aligned suggestions.
• Enhance educational coding assistants.

Topics

• Code Repair
• LLM Alignment
• Direct Preference Optimization
• Human-AI Teaming
• Software Engineering
• Feedback Generation

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Software Engineer

Related on AIssential

• Drifting Preference Optimization for One-Step Generative Models — DrPO enables efficient preference finetuning for one-step generative models using black-box rewards and feature-space updates.
• PracRepair: LLM-Empowered Automated Program Repair Inspired by Human-Like Debugging Practices — PracRepair enhances LLM-based automated program repair by integrating dynamic execution and validation feedback, mimicking human debugging.
• AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completion — AlignCoder enhances RAG for repository-level code completion by using multiple candidate completions to refine queries and training a retriever with reinforcement learning.
• Recovering Diversity Without Losing Alignment: A DPO Recipe for Post-Trained LLMs — Post-training LLMs can regain output diversity without losing alignment by carefully constructing DPO preference data.
• SWE-IF: Aligning Code Evaluation with Human Preference — Human preference in code generation requires evaluating both functional correctness and instruction following, not just functionality.
• Preference tuning explained — Preference tuning teaches models to generate humanlike responses by learning from preferred answer pairs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.