When Reasoning Narrows the Move: Diversity Collapse in LLM Game Play

2026-07-23 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

The study "When Reasoning Narrows the Move: Diversity Collapse in LLM Game Play" investigates how Supervised Fine-Tuning (SFT) impacts behavioral diversity in large language models (LLMs) used for sequential decision-making. Researchers used a controlled suite of deterministic board games, including tic-tac-toe variants, and fine-tuned Qwen3-8B models. They found that reasoning-mode generation frequently suppresses action diversity without uniformly improving action accuracy, observed in 21 out of 24 scenarios across trained and new state pools. Standard SFT significantly improves accuracy but often induces premature diversity collapse, exceeding the minimal requirement of the accuracy–diversity tradeoff. Action augmentation, which trains on all optimal actions per state instead of a single one, partially mitigates this effect, preserving non-trivial diversity at competitive performance. This highlights narrow-support imitation as a source of policy collapse in LLM decision-making.

Key takeaway

For machine learning engineers developing LLM-based decision-making agents, you should critically evaluate the impact of Supervised Fine-Tuning (SFT) on policy diversity. If your application requires exploratory behavior or adaptability, consider implementing action augmentation during SFT by including all optimal actions per state. Relying solely on standard SFT or reasoning-mode generation risks premature diversity collapse, potentially limiting your agent's robustness and ability to discover optimal strategies in complex environments.

Key insights

SFT and reasoning in LLMs can prematurely collapse action diversity, even without significant accuracy gains.

Principles

• Reasoning mode can amplify weak action preferences.
• Standard SFT induces policy concentration beyond optimal play.
• Action augmentation preserves diversity during fine-tuning.

Method

The study involved fine-tuning Qwen3-8B on tic-tac-toe variants using SFT datasets varying in action support (single vs. all optimal actions) and output format (direct vs. reasoning trace), then evaluating state-level accuracy and action/trajectory diversity.

In practice

• Consider action augmentation for SFT datasets.
• Evaluate diversity alongside accuracy in LLM policies.
• Be wary of reasoning mode's diversity impact.

Topics

• Large Language Models
• Supervised Fine-Tuning
• Action Diversity
• Game AI
• Policy Collapse
• Qwen3-8B

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• When Reasoning Narrows the Move: Diversity Collapse in LLM Game Play — SFT can prematurely collapse LLM action diversity in sequential decision-making, even with accuracy gains.
• Fine-Tuning LLMs, Part 2: SFT, LoRA, DPO, and the RL Methods That Power Reasoning Models — Fine-tuning and reinforcement learning are complementary stages for LLM specialization, not competing choices.
• R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training — Diversity Illusion hinders LLM self-play, where training data appears diverse but lacks true underlying skill variation.
• Reasoning Fine-Tuning Induces Persistent Latent Policy States — Reasoning fine-tuning globally reorganizes language model latent dynamics, enabling process-level control and performance gains.
• d’Olle Grieze at SemEval-2026 Task 11: Comparing the Impact of Supervised Fine-Tuning and Activation Steering on Mitigating Content Effect Bias in Syllogistic Reasoning — Supervised fine-tuning effectively mitigates content effect bias in LLMs by shifting focus to logical structure, outperforming activation steering.
• The Evolution of Reasoning in Small Language Models with Yejin Choi - #761 — High-quality, diverse data and integrated learning approaches are key to enhancing small language model reasoning.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.