The Death of RLHF: A Practitioner’s Guide to the New Post-Training Stack

2026-04-17 · Source: Towards AI - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, long

Summary

The traditional post-training pipeline for large language models (LLMs), which relied on Supervised Fine-Tuning (SFT) followed by Reinforcement Learning from Human Feedback (RLHF), is now largely obsolete. Modern reasoning models like DeepSeek-R1, Nemotron 3 Super, and Qwen3 utilize a new modular stack that addresses RLHF's limitations: annotation bottlenecks, high computational cost due to a four-model setup, and reward model drift. Key innovations include Group Relative Policy Optimization (GRPO), which eliminates the critic and reward model by using group statistics for advantage calculation, making RL feasible on a single GPU. Reinforcement Learning with Verifiable Rewards (RLVR) replaces human judgment with programmatic verifiers for tasks like math and code, offering scalable and consistent rewards. Decoupled clip and Dynamic sAmpling Policy Optimization (DAPO) further stabilizes GRPO for long chain-of-thought outputs by fixing length bias, entropy collapse, and vanishing gradients. This new stack separates instruction following (SFT), alignment (DPO/SimPO), and reasoning (GRPO+RLVR).

Key takeaway

For AI Engineers fine-tuning LLMs for reasoning tasks, the shift from RLHF to GRPO+RLVR with DAPO fixes means you can now achieve advanced reasoning capabilities on more modest hardware. You should prioritize developing robust programmatic verifiers for your domain-specific tasks, as these are now the primary driver of model improvement. Be mindful of potential issues like reward hacking and training instabilities with long chain-of-thought outputs, and configure your training with token-level loss and appropriate clipping bounds.

Key insights

New post-training methods like GRPO, RLVR, and DAPO have replaced RLHF for reasoning tasks, improving scalability and efficiency.

Principles

• Eliminate human annotation bottlenecks.
• Reduce computational overhead in RL.
• Modularize post-training objectives.

Method

The new stack involves SFT for instruction following, DPO/SimPO for alignment, and GRPO+RLVR (with DAPO fixes) for reasoning, using programmatic verifiers for rewards.

In practice

• GRPO enables RL fine-tuning on consumer GPUs.
• Programmatic verifiers are the new reward function lever.
• DAPO fixes GRPO instabilities for long CoT outputs.

Topics

• LLM Post-Training Stack
• Group Relative Policy Optimization
• Reinforcement Learning with Verifiable Rewards
• Decoupled clip and Dynamic sAmpling Policy Optimization
• Programmatic Verifiers

Best for: AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Post-Training Matters More Than Pretraining Now: SFT, RLHF, DPO, and GRPO. — Post-training methods are now critical for advancing LLM capabilities beyond mere imitation to complex reasoning.
• GRPO++: Tricks for Making RL Actually Work — Optimizing GRPO for LLM reasoning requires addressing inherent biases and system-level mismatches to achieve stable and efficient training.
• Reasoning RL in 2026: GRPO, DPO, RLVR, Agentic PO & Beyond — Modern reasoning RL prioritizes cheaper, critic-free, and preference-based methods over traditional PPO-style pipelines.
• Efficient Exploration, Reasoning, and Training-Free MTP — Efficiency gains in LLMs are achievable through online RLHF, balanced reasoning control, and embedding-space multi-token prediction.
• sGPO: Trading Inference FLOPs for Training Efficiency in RLVR — sGPO uses cheap inference to profile query difficulty, optimizing RLVR training by reducing wasted FLOPs.
• Dynamic Rollout Editing for Reducing Overthinking in RL-Trained Reasoning Models — Dynamic Rollout Editing (DRE) reduces LLM overthinking in RL training by editing successful reasoning rollouts to weaken unnecessary continuation signals.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards AI - Medium.