When Reasoning Narrows the Move: Diversity Collapse in LLM Game Play

2026-07-21 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

Research explores how Supervised Fine-Tuning (SFT) impacts behavioral diversity in large language models (LLMs) performing sequential decision-making tasks. Using deterministic board games, including tic-tac-toe variants, the study found that reasoning-mode generation often suppresses action diversity without consistently improving accuracy. Standard SFT, while enhancing accuracy, frequently induces a premature diversity collapse that exceeds the necessary accuracy-diversity tradeoff. To mitigate this, the authors propose action augmentation, a method that trains on all optimal actions per state rather than a single demonstrated action. This work identifies narrow-support imitation as a key cause of policy collapse in LLM decision-making, emphasizing the importance of preserving action support during SFT for maintaining exploratory behavior.

Key takeaway

For Machine Learning Engineers fine-tuning LLMs for sequential decision tasks, be aware that standard Supervised Fine-Tuning (SFT) can prematurely reduce action diversity, even if accuracy improves. You should consider implementing action augmentation, which trains on all optimal actions per state, to preserve exploratory behavior and prevent policy collapse. Prioritize evaluating diversity metrics alongside accuracy to ensure robust model performance in game play or similar environments.

Key insights

SFT can prematurely collapse LLM action diversity in sequential decision-making, even with accuracy gains.

Principles

• Reasoning-mode generation suppresses action diversity.
• Standard SFT often induces premature diversity collapse.
• Preserving action support during SFT maintains exploratory behavior.

Method

Action augmentation trains on all optimal actions per state, rather than a single demonstrated action, to mitigate diversity collapse during SFT.

In practice

• Implement action augmentation in SFT.
• Evaluate diversity alongside accuracy.
• Train on multi-optimal actions per state.

Topics

• Large Language Models
• Supervised Fine-tuning
• Behavioral Diversity
• Sequential Decision-Making
• Action Augmentation
• Policy Collapse

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• When Reasoning Narrows the Move: Diversity Collapse in LLM Game Play — SFT and reasoning in LLMs can prematurely collapse action diversity, even without significant accuracy gains.
• A Close Look At World Model Recovery In Supervised Fine-Tuned LLM Planners — SFT enables LLMs to internally represent planning world models, with broader training data improving recovery.
• Continual Learning with RL for LLMs — On-policy Reinforcement Learning inherently mitigates catastrophic forgetting and enhances generalization in LLMs, unlike Supervised Finetuning.
• Diverse reasoning traces teach LLMs to make better decisions — Training LLMs with diverse reasoning traces and explicit mode selection tokens improves accuracy and prevents mode collapse.
• Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution — Vocabulary dropout prevents co-evolutionary LLM collapse by diversifying problem generation through random output logit masking.
• Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RL — A unifying framework decomposes RL advantage functions to dynamically adapt gradient weighting, improving LLM reasoning.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.