Unified Data Selection for LLM Reasoning

2026-05-21 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

A new training-free metric, High-Entropy Sum (HES), addresses the challenge of selecting high-quality reasoning data for Large Language Models (LLMs) engaged in complex, long-CoT reasoning. HES quantifies reasoning quality by summing the entropy of only the top 0.5% highest-entropy tokens within each reasoning sample. Validated across Supervised Fine-tuning (SFT), Rejection Fine-tuning (RFT), and Reinforcement Learning (RL), HES consistently proves effective and computationally efficient. In SFT, using the top 20% HES-ranked data achieves performance comparable to training on the full dataset, while low-HES data degrades it. For RFT, HES-based training significantly outperforms baseline methods. In RL, HES-selected successful trajectories enable strong reasoning patterns, surpassing other approaches. This establishes HES as a robust, unified method for developing advanced LLM reasoning.

Key takeaway

For Machine Learning Engineers optimizing LLM training for complex reasoning, High-Entropy Sum (HES) offers a significant efficiency gain. You can achieve full-dataset Supervised Fine-tuning performance by selecting just the top 20% HES-ranked data, drastically reducing computational costs. Consider integrating HES into your Rejection Fine-tuning and Reinforcement Learning pipelines to reliably identify and utilize high-quality reasoning samples, accelerating model development and improving reasoning capabilities.

Key insights

High-Entropy Sum (HES) is a training-free metric for quantifying LLM reasoning quality via token entropy.

Principles

• High-entropy tokens indicate reasoning quality
• Data selection can match full-dataset performance
• Training-free metrics reduce computational overhead

Method

HES calculates reasoning quality by summing the entropy of the top 0.5% highest-entropy tokens in a reasoning sample.

In practice

• Train SFT models on top 20% HES data
• Apply HES for RFT data selection
• Use HES to select successful RL trajectories

Topics

• Large Language Models
• Data Selection
• Reasoning
• Supervised Fine-tuning
• Rejection Fine-tuning
• Reinforcement Learning
• High-Entropy Sum

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Unified Data Selection for LLM Reasoning — High-Entropy Sum (HES) quantifies LLM reasoning quality by focusing on cumulative uncertainty at critical decision points.
• Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning — Larger LLMs demonstrate superior efficiency and performance in RL post-training for mathematical reasoning.
• Revisiting the Uniform Information Density Hypothesis in LLM Reasoning — Uniform Information Density (UID) metrics predict LLM reasoning quality, especially local uniformity with global non-uniformity.
• Scaling LLM Reasoning from Minimal Labels: A Semi-Supervised Framework with a Lightweight Verifier — A semi-supervised framework scales LLM reasoning by using a lightweight verifier to generate high-confidence pseudo-labels from minimal human input.
• Entropy Centroids as Intrinsic Rewards for Test-Time Scaling — Temporal patterns of LLM uncertainty, captured by Entropy Centroids, predict response quality without external rewards.
• HARP: Efficient Data Selection for Finetuning Large Language Models — HARP efficiently selects finetuning data for LLMs by hierarchically evaluating representative subsets and inferring utilities.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.