Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability

2026-04-10 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new study challenges the common belief that Supervised Fine-Tuning (SFT) primarily leads to memorization, while Reinforcement Learning (RL) fosters generalization in Large Language Models (LLMs). Researchers found that cross-domain generalization in reasoning SFT, particularly with long Chain-of-Thought (CoT) supervision, is conditional. This generalization is influenced by optimization dynamics, training data quality, and the base model's inherent capability. The study observed a "dip-and-recovery" pattern where cross-domain performance initially degrades before improving with extended training, suggesting that short training periods can misrepresent generalization. High-quality, verified long-CoT traces consistently improved cross-domain reasoning, while low-quality solutions were detrimental. Stronger models could internalize transferable procedural patterns, even from simple tasks, unlike weaker models that merely mimicked surface-level verbosity. However, this generalization is asymmetric, improving reasoning but degrading safety.

Key takeaway

For AI Engineers and Research Scientists evaluating LLM post-training strategies, understand that reasoning SFT can achieve cross-domain generalization, but it requires careful consideration of training duration and data quality. Do not prematurely conclude SFT failures based on early training checkpoints, as performance may recover. Focus on curating high-quality, long Chain-of-Thought data and leverage more capable base models to foster robust reasoning generalization, while also monitoring for potential safety degradation.

Key insights

Reasoning SFT can generalize cross-domain, but it is conditional on optimization, data quality, and model capability.

Principles

• Generalization can exhibit a "dip-and-recovery" pattern.
• High-quality CoT data improves cross-domain reasoning.
• Stronger models internalize transferable reasoning patterns.

In practice

• Extend SFT training to overcome initial performance dips.
• Prioritize verified, long CoT traces for training data.
• Use stronger base models for better reasoning generalization.

Topics

• Reasoning SFT
• Cross-domain Generalization
• Chain-of-Thought
• Optimization Dynamics
• Model Capability

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• How Post-Training Shapes Biological Reasoning Models — Post-training stages distinctly shape biological reasoning model generalization, requiring careful composition for optimal performance.
• DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoning — Difficulty-aware data partitioning for SFT-then-RL training significantly enhances LLM general reasoning.
• Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning — Larger LLMs demonstrate superior efficiency and performance in RL post-training for mathematical reasoning.
• From Shortcuts to Reasoning: Robust Post-Training of Theory of Mind with Reinforcement Learning — Thinking-RFT, combining reasoning and RL, robustly improves Theory of Mind in foundation models by avoiding dataset shortcuts.
• Beyond KV-Cache: Test-Time Training & Invariant Latent Topologies for ICL — New methods enable LLMs to learn during inference and generalize across diverse domains by modifying internal weight structures or identifying invariant neural subspaces.
• Learning What to Learn: Stage-Specific Data Sets for SFT-then-RL in Small Language Model Reasoning — Data strategy for SFT-then-RL pipelines should align with each stage's distinct role in skill acquisition and consolidation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.