PERSA: Reinforcement Learning for Professor-Style Personalized Feedback with LLMs

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, quick

Summary

PERSA is a Reinforcement Learning from Human Feedback (RLHF) pipeline designed to adapt large language models (LLMs) for generating personalized, professor-style programming feedback. It addresses the challenge of aligning an LLM's stylistic tone with a specific instructor while maintaining diagnostic correctness. PERSA combines supervised fine-tuning on professor demonstrations, reward modeling from pairwise preferences, and Proximal-based policy optimization. Critically, it employs parameter-efficient fine-tuning, updating only the top transformer blocks and their feed-forward projections to minimize global parameter drift and increase stylistic controllability. Evaluated on APPS, PyFiXV, and CodeReviewQA benchmarks using Llama-3 and Gemma-2 backbones, PERSA achieved a Style Alignment Score (SAC) of 96.2% (from 34.8% for Base) and 100% Correctness Accuracy (CA) on APPS, demonstrating strong style transfer without sacrificing correctness.

Key takeaway

For Machine Learning Engineers developing educational AI tools, PERSA offers a practical method to align LLM output with specific pedagogical styles. You can achieve high style alignment, boosting Style Alignment Score to 96.2% while maintaining 100% Correctness Accuracy, by implementing parameter-efficient RLHF techniques. Consider applying similar constrained fine-tuning approaches to customize feedback tone without compromising factual accuracy in your applications.

Key insights

PERSA uses RLHF and parameter-efficient fine-tuning to align LLM feedback style with instructors while preserving correctness.

Principles

• Constrain learning to style-bearing components for targeted adaptation.
• Update only top transformer blocks for efficient style control.
• Combine SFT, reward modeling, and PPO for robust style transfer.

Method

PERSA's RLHF pipeline integrates supervised fine-tuning on professor demonstrations, reward modeling from pairwise preferences, and Proximal-based policy optimization, focusing parameter updates on top transformer blocks.

In practice

• Generate automated programming feedback with instructor tone.
• Adapt LLMs for personalized educational feedback.
• Apply parameter-efficient fine-tuning for style alignment.

Topics

• Reinforcement Learning from Human Feedback
• Large Language Models
• Parameter-Efficient Fine-Tuning
• Educational Technology
• Automated Feedback Generation
• Llama-3
• Gemma-2

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Reinforcement Learning with Human Feedback (RLHF), Clearly Explained!!! — RLHF aligns LLMs to human preferences by training a reward model from comparative feedback, then using it to fine-tune the LLM.
• OPERA: Aligning Open-Ended Reasoning via Objective Perplexity-based Reinforcement Learning — OPERA aligns LLMs for open-ended tasks using intrinsic perplexity-based rewards, overcoming external judge biases.
• Fine-tuning Multi-modal LLMs with ART: Art-based Reinforcement Training — ART fine-tunes MLLMs by optimizing raw visual input, enabling soft-token methods on frozen, pre-compiled models without graph modification.
• FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Users — FSPO enables LLMs to personalize responses by meta-learning user preferences from few-shot synthetic data.
• Learning to Prompt: Improving Student Engagement with Adaptive LLM-based High-School Tutoring — Adaptive prompt routing with subject-aware LLM evaluation improves tutoring efficiency and student engagement in diverse subjects.
• Steering at the Source: Style Modulation Heads for Robust Persona Control — Targeting specific "Style Modulation Heads" in LLMs enables robust persona control with minimal coherency degradation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.