Why Does RL Generalize Better Than SFT? A Data-Centric Perspective on VLM Post-Training

2026-02-11 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

Large-scale Vision-Language Models (VLMs) post-trained with Reinforcement Learning (RL) exhibit superior out-of-distribution (OOD) generalization compared to those trained with Supervised Fine-Tuning (SFT). This phenomenon, observed on February 11, 2026, is attributed to RL's implicit data filtering, which prioritizes medium-difficulty training samples. Research systematically evaluated SFT models across varying data difficulty levels, confirming that training on hard samples significantly degrades OOD performance. Based on this finding, a new method called Difficulty-Curated SFT (DC-SFT) was introduced. DC-SFT explicitly filters training data by sample difficulty, demonstrating substantial improvements in OOD generalization over standard SFT and even surpassing RL-based training, while offering enhanced stability and computational efficiency. Code for DC-SFT is available on GitHub.

Key takeaway

For research scientists and VLM engineers optimizing model generalization, consider implementing Difficulty-Curated SFT (DC-SFT) in your post-training pipeline. This method, which explicitly filters training data by difficulty, offers a more stable and computationally efficient path to superior out-of-distribution performance than traditional SFT or even RL-based approaches. You should evaluate your training data's difficulty distribution and curate it to prioritize medium-difficulty samples.

Key insights

RL's OOD generalization advantage in VLMs stems from implicitly prioritizing medium-difficulty training data.

Principles

• Hard training samples degrade OOD performance.
• Data difficulty is a critical generalization factor.

Method

Difficulty-Curated SFT (DC-SFT) explicitly filters VLM training data based on sample difficulty to enhance OOD generalization, outperforming standard SFT and RL.

In practice

• Filter training data by difficulty for better VLM generalization.
• Prioritize medium-difficulty samples in VLM fine-tuning.

Topics

• Vision-Language Models
• Reinforcement Learning
• Supervised Fine-Tuning
• OOD Generalization
• Data Filtering

Code references

• byyx666/DC-SFT

Best for: Computer Vision Engineer, Research Scientist, AI Researcher, AI Scientist, Machine Learning Engineer

Related on AIssential

• DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoning — Difficulty-aware data partitioning for SFT-then-RL training significantly enhances LLM general reasoning.
• Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability — Reasoning SFT can generalize cross-domain, but it is conditional on optimization, data quality, and model capability.
• Efficient Data Selection for Multimodal Models via Incremental Optimization Utility — One-Step-Train (OST) optimizes LMM data selection by estimating marginal utility via a lightweight proxy, improving efficiency and performance.
• Learning What to Learn: Stage-Specific Data Sets for SFT-then-RL in Small Language Model Reasoning — Data strategy for SFT-then-RL pipelines should align with each stage's distinct role in skill acquisition and consolidation.
• Meta-Learning Transformers to Improve In-Context Generalization — Training transformers on diverse small datasets improves in-context generalization and modularity over large, uncurated corpora.
• Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently — RLVR's outcome-based rewards enable LLMs to learn efficient backtracking, unlike SFT, leading to exponential inference compute gains.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.