One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models

2026-01-16 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

Reward Models (RMs), crucial for aligning language models (LMs) with human preferences, are shown to still suffer from persistent biases like length, sycophancy, and overconfidence, even in state-of-the-art models. This research also identifies novel issues, including bias towards model-specific "styles" and answer-order (position bias). To address low-complexity biases arising from spurious correlations, the authors propose a "mechanistic reward shaping" intervention using linear activation probes. This data-efficient, model-internal method effectively reduces targeted biases (length, uncertainty, position) without degrading overall reward quality and generalizes out-of-distribution, as validated on RewardBench-2. However, high-complexity biases like sycophancy and model-style sensitivity remain resistant to simple linear interventions, indicating a need for more sophisticated solutions.

Key takeaway

Reward Models (RMs) for LLM alignment exhibit persistent biases (length, sycophancy, overconfidence) and new ones (model-specific styles, answer-order). A mechanistic reward shaping approach, using linear activation probes constructed via difference-of-means, effectively mitigates low-complexity biases like length, uncertainty, and position across five RMs, including SOTA, without degrading RewardBench-2 accuracy. This data-efficient, model-internal intervention offers a practical post-hoc solution for improving RM robustness, though complex biases like sycophancy remain challenging.

Topics

• Reward Models
• Language Model Biases
• Reinforcement Learning from Human Feedback
• Linear Probes
• Mechanistic Interpretability

Code references

• drfein/OneBiasAfterAnother

Best for: Research Scientist, AI Researcher, AI Scientist, NLP Engineer

Related on AIssential

• Beyond Distribution Sharpening: The Importance of Task Rewards — Task-reward optimization in LLM fine-tuning offers stable, superior performance over distribution sharpening, especially for complex tasks.
• That’s Not Alignment. It’s Formatting Overfitting. — Reward models can overfit to prompt formatting, creating an illusion of alignment without true behavioral change.
• Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax — Semantic-space alignment with reinforcement learning mitigates "alignment tax" in low-resource language expansion for LLMs.
• In-Context Reward Adaptation for Robust Preference Modeling — In-Context Reward Adaptation uses transformers and human response time to dynamically model diverse, unseen human preferences for robust RLHF.
• A Pre-Registered Causal Partition of Self-Consistency Elicitation and Reward Design in RLVR — The Δ_naive estimand in RLVR conflates self-consistency elicitation with genuine reward design, leading to biased attribution.
• BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization — BiasGRPO stabilizes LLM bias mitigation by normalizing rewards across completion groups, addressing high-variance subjective landscapes.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.