Fine-Tuning LLMs, Part 2: SFT, LoRA, DPO, and the RL Methods That Power Reasoning Models

2026-07-21 · Source: AI Advances - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, extended

Summary

This article details advanced fine-tuning techniques for Large Language Models (LLMs), focusing on Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and online Reinforcement Learning (RL) methods such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO). It explains how these methods specialize pre-trained models for specific tasks, contrasting imitation-based SFT with feedback-driven RL. The DeepSeek-R1 model is presented as a real-world example, demonstrating a staged pipeline combining initial SFT with subsequent GRPO for enhanced reasoning, achieving over 70% on the AIME maths benchmark. The piece also covers instruction fine-tuning, the role of chat templates, and practical considerations for SFT, including data quality, hardware (e.g., 16GB VRAM GPU, Google Colab T4), hyperparameters (learning rate 5e-5 to 1e-4, batch size 4-16), and monitoring metrics like training and validation loss.

Key takeaway

For AI Engineers fine-tuning LLMs, understand that Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are sequential stages. You should first apply SFT with high-quality, domain-specific data to establish desired format and tone, then consider RL methods like GRPO or PPO to refine complex reasoning or align with fuzzy preferences. Prioritize data quality over quantity and remember fine-tuning teaches behavior, not new facts; integrate RAG for fresh knowledge.

Key insights

Fine-tuning and reinforcement learning are complementary stages for LLM specialization, not competing choices.

Principles

• Quality data is paramount for SFT, not quantity.
• RL can generate novel behaviors beyond human examples.
• Fine-tuning teaches behavior, not new factual knowledge.

Method

The SFT process involves data preparation (1,000+ high-quality examples), selecting hardware (16GB VRAM GPU), configuring hyperparameters (learning rate, batch size, sequence length, steps), and monitoring training/validation loss and sample outputs.

In practice

• Use "add_generation_prompt=True" for inference.
• Use "add_generation_prompt=False" for training data.
• Track metrics like loss, learning rate, and dataset characteristics.

Topics

• LLM Fine-tuning
• Supervised Fine-tuning
• Reinforcement Learning
• DeepSeek-R1
• Chat Templates
• Hyperparameter Optimization

Best for: AI Engineer, Machine Learning Engineer, AI Scientist

Related on AIssential

• Select and Improve: Understanding the Mechanics of Post-Training for Reasoning — RL post-training enhances reasoning via strategy selection and improvement, driven by diverse pre-training and challenging RL data.
• How to Fine-Tune an LLM: SFT, LoRA, QLoRA and DPO Explained — Fine-tuning adapts LLMs for specific tasks and behaviors using SFT, LoRA, QLoRA, and DPO.
• Training Agents: Live tutorial on how to fine-tune a coding agent for continual learning — The tutorial outlines a supervised fine-tuning (SFT) baseline for coding agents using public traces and TRL + LoRA.
• AI 101: Beyond RL: The New Fine-Tuning Stack for LLMs — Modern LLM fine-tuning is evolving into a modular stack, moving beyond monolithic RL to integrate diverse, efficient methods.
• Why Fine-Tuned LLMs (SFT & LoRA) Fail to Reason — Fine-tuned LLMs memorize facts but fail to reason with them due to internal communication breakdowns, not storage.
• Understanding post-training of LLMs: SFT — Post-training guides LLM behavior and expression, not knowledge, primarily via SFT and preference optimization.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI Advances - Medium.