How LLMs Learn to Be Helpful (RLHF vs DPO)

2025-12-15 · Source: ByteByteGo Newsletter · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Intermediate, long

Summary

Large Language Models (LLMs) learn helpfulness through a three-stage pipeline: pretraining, supervised fine-tuning (SFT), and preference learning. SFT teaches instruction following but struggles with questions having multiple good answers. The third stage, preference learning, uses human comparisons of responses to teach models trade-offs. Two main methods are Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO). RLHF trains a separate reward model and uses PPO to optimize the policy, involving four models and being complex. DPO simplifies this by folding the reward signal into a single training step, directly adjusting the model's probabilities. For example, Zephyr, a 7 billion parameter DPO model, outperformed Llama 2 Chat 70B. Both RLHF and DPO face "reward hacking" where models exploit proxy signals, leading to issues like sycophancy. Verifiable rewards, used for tasks with checkable answers like math or code, offer an exact signal, bypassing proxy problems.

Key takeaway

For Machine Learning Engineers developing helpful LLMs, understanding the trade-offs between RLHF and DPO is crucial. While RLHF is powerful but complex, DPO offers a simpler, more cost-effective approach for preference learning, making alignment accessible. However, you must mitigate "reward hacking" and sycophancy by carefully evaluating proxy signals. For tasks with verifiable answers, prioritize methods using exact, programmatic rewards to avoid these issues.

Key insights

LLMs learn helpfulness via preference learning, primarily through RLHF or the simpler DPO, to navigate subjective trade-offs in responses.

Principles

• Human judgment is easier for comparison than creation.
• Alignment can outweigh model size (InstructGPT vs. GPT-3).
• Proxy rewards can lead to "reward hacking."

Method

LLM helpfulness training involves pretraining, supervised fine-tuning (SFT) for instruction following, and a third stage for preference learning using human comparisons via RLHF or DPO.

In practice

• Use DPO for cost-effective alignment on modest budgets.
• Apply verifiable rewards for tasks with objective answers.
• Be aware of sycophancy in models trained with proxy rewards.

Topics

• Large Language Models
• Reinforcement Learning from Human Feedback
• Direct Preference Optimization
• Supervised Fine-Tuning
• Reward Hacking
• Verifiable Rewards

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Align Large Language Model with Human Preference — RLHF aligns LLMs with human values for helpful, honest, and harmless text generation by leveraging human feedback.
• Understanding post-training of LLMs: DPO — DPO directly optimizes LLMs for human preferences without complex reward models or reinforcement learning loops.
• Reward hacking in Reinforcement learning — Reward hacking is an inevitable outcome of imperfect objective functions in RL, especially with powerful LLM optimizers.
• Rewarding Better Thinking for LLM Preference Alignment — Thinking Checklist Reward (TCR) improves LLM alignment by guiding reasoning trajectories with process-oriented, sample-specific checklists and a complementary "thinking surplus".
• Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment — P-GRPO enhances LLM alignment by normalizing advantages against preference-group-specific reward histories.
• The Death of RLHF: A Practitioner’s Guide to the New Post-Training Stack — New post-training methods like GRPO, RLVR, and DAPO have replaced RLHF for reasoning tasks, improving scalability and efficiency.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by ByteByteGo Newsletter.