Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning

2025-09-23 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, long

Summary

An empirical study investigated the scaling behaviors of Large Language Models (LLMs) during reinforcement learning (RL) post-training, specifically focusing on mathematical reasoning. Based on 54 experiments using the Qwen2.5 model family (0.5B to 14B parameters) and 50k mathematics problems, the research characterized how model scale, data volume, and computational budget interact to influence performance. Key findings include that larger models, even with fewer training steps, consistently outperform smaller models under fixed computational budgets. Larger models also exhibit superior sample efficiency with fixed data. In data-constrained scenarios, repeated reuse of high-quality data is highly effective, as performance is primarily driven by total optimization steps rather than unique samples. These scaling behaviors are robust across both base and instruction-tuned models, showing similar learning dynamics despite differences in absolute accuracy. The study also explored in-domain and out-of-domain generalization, noting positive transfer for other math tasks but limited or negative transfer for tasks like code generation or logical reasoning.

Key takeaway

For AI Engineers optimizing LLMs for mathematical reasoning, prioritize larger models (e.g., 14B parameters) even if it means fewer training steps, as they offer superior computational and data efficiency. In situations with limited unique training data, implement aggressive data reuse strategies, as total optimization steps are more critical than sample uniqueness. Be aware that specialized mathematical reasoning gains may not transfer positively to other domains like code generation or general logical reasoning, potentially even causing negative transfer in some cases.

Key insights

Larger LLMs demonstrate superior efficiency and performance in RL post-training for mathematical reasoning.

Principles

• Larger models yield better compute and data efficiency.
• Data reuse is effective in data-limited settings.
• Scaling behaviors are robust across model types.

Method

The study fine-tuned 54 LLMs using Group Relative Policy Optimization (GRPO) on 50k math problems, systematically varying model size, data volume, and computational budget to analyze performance.

In practice

• Prioritize larger models for RL post-training.
• Reuse high-quality data in data-scarce environments.
• Expect limited transfer to non-mathematical domains.

Topics

• LLM Scaling Laws
• Reinforcement Learning Post-Training
• Mathematical Reasoning
• Data Reuse Strategy
• Computational Budget Optimization

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Director of AI/ML

Related on AIssential

• A Quantitative Experimental Repeated Measures Study of Training Dynamics in a Small Llama Style Language Model Under a Compute-Aware Token Budget — Compute-constrained LLM training trajectories reveal non-monotonic degradation and instability, challenging endpoint-only evaluation.
• Adaptive Test-Time Compute Allocation for Reasoning LLMs via Constrained Policy Optimization — Optimizing LLM test-time compute involves balancing accuracy and cost via a constrained policy.
• AI Trends 2026: OpenClaw Agents, Reasoning LLMs, and More [Sebastian Raschka] - 762 — LLM evolution focuses on post-training reasoning, tool use, and inference scaling to enhance problem-solving and efficiency.
• RL Scaling Laws for LLMs — Scaling laws, while precise for LLM pretraining, are more complex and context-dependent in reinforcement learning.
• Stop Making Models Bigger, Make Them Behave — Kobie Crawdord, Snorkel — Focused RL training with high-quality, behavior-specific data enables smaller models to surpass larger ones in tool-use tasks.
• Making LLMs Think Longer: Context, State, and Post-Training Tricks — LLM reasoning tokens are externalized computational state, not transparent explanations of internal processes.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.