Unified Data Selection for LLM Reasoning

2026-05-22 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

The paper introduces High-Entropy Sum (HES), a training-free metric designed to quantify reasoning quality in Large Language Models (LLMs) by summing the entropy of the top 0.5% highest-entropy tokens in a reasoning sample. HES addresses the bottleneck of needing massive high-quality reasoning data for complex, long-CoT tasks. Validated across Supervised Fine-tuning (SFT), Rejection Fine-tuning (RFT), and Reinforcement Learning (RL) paradigms, HES consistently improves performance and reduces computational overhead. In SFT, training on the top 20% HES-ranked data matches full-dataset performance, while using a lightweight 0.6B model for selection yields results comparable to an 8B model. HES also significantly outperforms baselines in RFT and enables stronger reasoning patterns in RL by selecting successful trajectories.

Key takeaway

For MLOps engineers optimizing LLM training pipelines, integrating High-Entropy Sum (HES) for data selection offers significant efficiency gains. You can achieve comparable or superior model performance by training on just the top 20% of HES-ranked data, drastically cutting computational costs. Consider using smaller proxy models (e.g., 0.6B) for offline HES calculation to further reduce inference overhead during data curation. This approach ensures high-quality reasoning data without extensive resource investment.

Key insights

High-Entropy Sum (HES) quantifies LLM reasoning quality by focusing on cumulative uncertainty at critical decision points.

Principles

• High-entropy tokens signal critical decision points in reasoning paths.
• Cumulative entropy of critical tokens correlates with reasoning quality.
• Small proxy models can efficiently select data for larger LLMs.

Method

HES is calculated by summing entropy values of the top $k\%$ (e.g., 0.5%) highest-entropy tokens in a reasoning sample. This metric guides data selection for SFT, RFT, and RL.

In practice

• Use HES to select top 20% data for SFT to match full-dataset performance.
• Apply HES as a training-free reward signal in RFT.
• Prioritize high-HES positive samples in RL for stronger reasoning.

Topics

• Large Language Models
• Chain-of-Thought Reasoning
• Data Selection
• Supervised Fine-tuning
• Rejection Fine-tuning
• Reinforcement Learning
• Token Entropy

Code references

• huggingface/open-r1

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Unified Data Selection for LLM Reasoning — High-Entropy Sum (HES) is a training-free metric for quantifying LLM reasoning quality via token entropy.
• Reasoning Quality Emerges Early: Data Curation for Reasoning Models — Reasoning quality in LLMs can be effectively improved by curating SFT data based on early token loss patterns, significantly boosting efficiency.
• Beyond Entropy: Learning from Token-Level Distributional Deviations for LLM Reasoning — ICT stabilizes LLM reasoning by focusing on token-level distributional deviations, preventing entropy collapse and explosion.
• Dynamic Sampling that Adapts: Self-Aware Iterative Data Persistent Optimization for Mathematical Reasoning — Dynamically adapting training data to a model's evolving capabilities significantly boosts mathematical reasoning performance.
• CAPS: Cascaded Adaptive Pairwise Selection for Efficient Parallel Reasoning — CAPS significantly reduces verifier-token cost in parallel reasoning by adaptively allocating compute for pairwise self-verification.
• Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning — Off-the-shelf LLMs can guide smaller models in mathematical reasoning by scoring fixed-length chunks, avoiding training and length bias.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.