Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging

2026-06-16 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Reinforcement Learning with Verifiable Reward (RLVR) is a post-training paradigm that excels over Supervised Fine-Tuning (SFT) in reasoning and resisting catastrophic forgetting. Recent findings indicate RLVR induces sparse, off-principal parameter updates, prompting an investigation into its amenability to model merging. Contrary to expectations, a "sparsity curse" was discovered: RLVR's sparse updates are widely dispersed in parameter space, creating near-orthogonal shortcuts that hinder aggregation and cause severe degradation with standard merging methods. This fragility stems from RL optimization stochasticity and diverse reasoning patterns, unlike SFT models that converge to shared, flat basins. The research characterizes these failure mechanisms and introduces Sensitivity-aware Resolving Merging (SAR-Merging). This new recipe resolves update conflicts using Fisher Information-based sensitivity arbitration, followed by magnitude-aware sparsification and rescaling to protect reasoning pathways. Experiments on mathematical and coding benchmarks confirm SAR-Merging's superior performance for RLVR models, enabling both single-task improvement and multi-capability fusion.

Key takeaway

For Machine Learning Engineers working with Reinforcement Learning with Verifiable Reward (RLVR) models, recognize that standard model merging techniques will likely cause severe performance degradation due to RLVR's unique sparse parameter space. Instead, you should investigate and implement Sensitivity-aware Resolving Merging (SAR-Merging) to effectively combine RLVR models. This specialized approach enables robust multi-capability fusion and single-task enhancement, overcoming the "sparsity curse" and preserving critical reasoning pathways in your models.

Key insights

RLVR's sparse, dispersed parameter updates create a "sparsity curse" that prevents effective model merging, unlike SFT models.

Principles

• RLVR updates are sparse and off-principal.
• RLVR updates form near-orthogonal shortcuts.
• Standard merging severely degrades RLVR models.

Method

SAR-Merging resolves update conflicts using Fisher Information-based sensitivity arbitration, then applies magnitude-aware sparsification and rescaling to preserve fragile reasoning pathways.

In practice

• Enhance single-task RLVR performance.
• Fuse multiple RLVR capabilities.

Topics

• Reinforcement Learning
• Model Merging
• Sparsity Curse
• Parameter Space
• SAR-Merging
• Fisher Information

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models — Multi-task RLVR for LLMs offers comparable performance to model merging with significantly reduced computational cost.
• Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals — Rigid clipping in RLVR discards valuable near-boundary signals, causing instability; stochastic recovery improves performance.
• Understanding Diversity Collapse in RLVR via the Lens of Overtraining — Diversity collapse in RLVR is overtraining, concentrating probability mass on already solved problems.
• MARS: Margin-Aware Reward-Modeling with Self-Refinement — MARS improves reward modeling by adaptively augmenting ambiguous preference data to enhance model robustness.
• Post-Hoc Merging is Not Enough: Many-Shot Model Merging with Loss-Gap Balancing — Iterative, loss-aware many-shot model merging effectively mitigates task interference and information erasure in multi-task LLMs.
• [P] I trained Qwen2.5-1.5b with RLVR (GRPO) vs SFT and compared benchmark performance — RLVR significantly boosts math reasoning in LLMs, while SFT can degrade it by overriding pretrained knowledge.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.