To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models

2026-02-16 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing, Data Science & Analytics · Depth: Expert, extended

Summary

This study, named M2RL, systematically compares two multi-domain Reinforcement Learning with Verifiable Rewards (RLVR) paradigms for Large Language Models (LLMs): mixed multi-task RLVR and separate RLVR followed by model merging. Using the Qwen3-4B-Base model and open-source Nemotron 3 Nano datasets, experiments were conducted across math, coding, science, and instruction following domains. Key findings indicate that mixed multi-task RLVR achieves comparable performance to model merging while requiring only 33.2% of the GPU hours. The research reveals minimal inter-task interference and significant synergistic effects, particularly among reasoning-intensive domains. Analysis of weight space geometry, model prediction behavior, and information constraints shows that RLVR induces self-discrimination and that multi-task training fosters cross-domain synergy, enhancing overall evaluation accuracy.

Key takeaway

Research scientists developing multi-domain LLMs should consider adopting mixed multi-task Reinforcement Learning with Verifiable Rewards (RLVR). This approach offers performance comparable to separate training and model merging, but with substantially lower computational overhead (33.2% fewer GPU hours). You can achieve synergistic gains, especially across reasoning-intensive tasks, by leveraging this method, which also naturally enhances the model's self-discrimination capabilities. This strategy provides a more efficient path to developing general expert-level LLMs.

Key insights

Multi-task RLVR for LLMs offers comparable performance to model merging with significantly reduced computational cost.

Principles

• Reasoning-intensive domains exhibit synergistic effects in RLVR.
• RLVR naturally induces self-discrimination capabilities in LLMs.
• Policy neighborhoods can enhance multi-domain model performance.

Method

The study uses a simplified Supervised Fine-Tuning (SFT) then Reinforcement Learning (RL) pipeline, applying Group Relative Policy Optimization (GRPO) with a domain-routed reward function for multi-task RLVR.

In practice

• Consider mixed multi-task RLVR for multi-domain LLM training to save GPU hours.
• Prioritize process-based verification for logic-intensive tasks like math and coding.
• Utilize outcome-based verification for constraint-intensive tasks such as instruction following.

Topics

• Reinforcement Learning with Verifiable Rewards
• Multi-domain Learning
• Model Merging
• Multi-task Training
• Policy Neighborhoods

Code references

• mosAI25/M2RL
• THUDM/slime

Best for: Research Scientist, AI Researcher, AI Scientist, Deep Learning Engineer

Related on AIssential

• Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently — RLVR's outcome-based rewards enable LLMs to learn efficient backtracking, unlike SFT, leading to exponential inference compute gains.
• Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging — RLVR's sparse, dispersed parameter updates create a "sparsity curse" that prevents effective model merging, unlike SFT models.
• Connect the Dots: Training LLMs for Long-Lifecycle Agents with Cross-Domain Generalization Via Reinforcement Learning — Training LLMs with end-to-end RL for long-lifecycle agents fosters continuous learning and cross-domain generalization.
• Let LLMs Wander: Engineering RL Environments — Stefano Fiorucci — RL environments enable LLMs to learn complex behaviors through interactive feedback, surpassing limitations of static supervised data.
• Paper: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — DeepSeek R1 enhances LLM reasoning via pure reinforcement learning and rule-based rewards, fostering autonomous problem-solving without supervised data.
• LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking — RLVR can lead to reward hacking in LLMs by incentivizing shortcut strategies that exploit imperfect verifiers.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.