PEAR: Supervised Fine-Tuning Optimized for RL

2024-03-06 · Source: The Salt - Curated AI · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Advanced, short

Summary

Two recent studies explore critical aspects of large language model (LLM) training and optimization. The first, "Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning," introduces PEAR, a loss-reweighting scheme for supervised fine-tuning (SFT) that improves post-Reinforcement Learning (RL) performance. Experiments with Qwen and DeepSeek-distilled models on logic games and math benchmarks show that traditional SFT objectives often lead to suboptimal RL initialization, whereas PEAR consistently boosts post-RL outcomes. The second study, "On the Limits of Layer Pruning for Generative Reasoning in LLMs," investigates the impact of layer pruning on 7-8B instruction models like Llama 3.1 and Qwen 2.5. It reveals that while classification accuracy remains robust, generative reasoning tasks (math, code) suffer significant degradation even with minimal pruning. The authors propose Self-Generated Responses (SGR) for fine-tuning pruned models, which improves performance retention compared to using open-source data.

Key takeaway

For AI Engineers and Research Scientists developing or deploying LLMs, understand that standard SFT may not optimally prepare models for subsequent RL, necessitating techniques like PEAR. If you are considering layer pruning for model compression, be aware that generative reasoning capabilities (math, code) are highly susceptible to degradation, even at low pruning ratios. Employ Self-Generated Responses (SGR) as a recovery strategy to retain more performance in pruned models, especially for generative tasks.

Key insights

Optimizing SFT for RL requires aligning offline and online learning, while layer pruning severely impacts LLM generative reasoning.

Principles

• SFT objectives can hinder subsequent RL performance.
• Layer pruning disproportionately harms generative reasoning.
• Self-distillation can mitigate pruning's negative effects.

Method

PEAR uses token-level likelihood ratios and importance weighting during SFT to align the model with its own policy for better RL initialization. SGR fine-tunes pruned models on responses generated by the unpruned base model.

In practice

• Implement PEAR for SFT to enhance post-RL model performance.
• Exercise caution with layer pruning for generative LLMs.
• Use SGR for fine-tuning pruned models to recover reasoning.

Topics

• Supervised Fine-Tuning
• Reinforcement Learning
• Layer Pruning
• Generative Reasoning
• Model Distillation

Code references

• safal312/on-the-limits-of-layer-pruning

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Researcher, AI Scientist, Machine Learning Engineer

Related on AIssential

• Two-Stage Regularization-Based Structured Pruning for LLMs — Regularization before pruning can effectively transfer information, preserving LLM performance and reducing fine-tuning needs.
• Select and Improve: Understanding the Mechanics of Post-Training for Reasoning — RL post-training enhances reasoning via strategy selection and improvement, driven by diverse pre-training and challenging RL data.
• The Benchmark Illusion: Pruned LLMs Can Pass Multiple Choice but Fail to Answer — Pruned LLMs often recognize correct answers in multiple-choice but fail to generate them, revealing a benchmark illusion.
• Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy — A GEMM-centric taxonomy unifies LLM pruning evaluation, revealing practical acceleration limits and optimal strategies across quality loss levels.
• Hugging Face Journal Club: Embarrassingly Simple Self-Distillation Improves Code Generation — Self-distillation via temperature-tuned SFT improves model performance by refining token distribution.
• Cascaded Multi-Granularity Pruning for On-Device LLM Inference in Industrial IoT — Cascaded multi-granularity pruning with low-rank recovery enables extreme LLM compression for IIoT, guided by architectural independence.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by The Salt - Curated AI.