Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution

2026-04-03 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

Vocabulary dropout, a novel intervention, addresses the common failure of co-evolutionary self-play in large language models, where a problem-generating LLM converges to a narrow problem distribution, hindering the solver's learning. Researchers introduce this lightweight technique that randomly masks the proposer's output logits during both policy training and curriculum generation. This hard, non-stationary mask prevents the proposer from locking into fixed token sequences, thereby sustaining diversity. When applied to Qwen3-4B and Qwen3-8B models for mathematical reasoning using R-Zero, vocabulary dropout improved the solver's performance by an average of +4.4 points at 8B, with significant gains on competition-level benchmarks. This method demonstrates that explicit action-space constraints are vital for productive language co-evolution.

Key takeaway

For AI scientists and ML engineers developing self-play or co-evolutionary LLM systems, consider implementing vocabulary dropout to prevent curriculum collapse. This technique improved Qwen3-8B's mathematical reasoning by +4.4 points. It offers a simple yet effective way to impose necessary action-space constraints, ensuring sustained diversity in problem generation. Adopting this method can lead to more robust and capable LLMs, especially for complex reasoning tasks.

Key insights

Vocabulary dropout prevents co-evolutionary LLM collapse by diversifying problem generation through random output logit masking.

Principles

• Co-evolutionary LLMs need action-space constraints.
• Non-stationary masking diversifies LLM output.
• Curriculum diversity prevents learning stalls.

Method

Vocabulary dropout randomly masks the proposer's output logits during both policy training and curriculum generation. This hard, non-stationary mask prevents the proposer from generating fixed token sequences, sustaining diversity.

In practice

• Apply vocabulary dropout to LLM self-play.
• Enhance mathematical reasoning LLM performance.
• Improve curriculum diversity in generative models.

Topics

• LLM Co-evolution
• Vocabulary Dropout
• Curriculum Learning
• Self-Play
• Mathematical Reasoning
• Qwen3 Models

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills — Skill Self-Play uses co-evolving agent skills to balance task diversity and reliable verification in LLM self-evolution.
• Beyond Mode Collapse: Distribution Matching for Diverse Reasoning — Distribution matching prevents mode collapse in on-policy RL by promoting diverse exploration, improving reasoning capabilities.
• When Reasoning Narrows the Move: Diversity Collapse in LLM Game Play — SFT can prematurely collapse LLM action diversity in sequential decision-making, even with accuracy gains.
• Solving the Wrong Problem Works Better - Robert Lange — Shinka Evolve uses LLMs and evolutionary algorithms for sample-efficient program generation and scientific discovery.
• LLM-Evolved Domain-Independent Heuristics for Symbolic AI Planning — LLMs can evolve domain-independent heuristics for symbolic AI planning that outperform human-designed ones.
• Large Language Model-Driven Full-Component Evolution of Adaptive Large Neighborhood Search — LLM-driven evolution can automate and enhance complex metaheuristic algorithm design by decoupling and optimizing core components.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.