Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization

2026-04-16 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, long

Summary

Researchers from the University of Science and Technology of China and Meituan have developed Length Controlled Preference Optimization (LCPO), a method to reduce the lengthy Chain-of-Thought (CoT) outputs of Large Reasoning Models (LRMs) like Deepseek-R1-Distill-Qwen-1.5B and Deepseek-R1-Distill-Qwen-7B. LRMs often generate verbose responses, increasing computational costs and potentially leading to "overthinking" on simpler tasks. LCPO addresses this by analyzing generation path distributions and filtering trajectories based on difficulty estimation. It then applies a small-scale preference optimization, balancing the implicit reward related to Negative Log-Likelihood (NLL) loss. Experiments across six math reasoning benchmarks, including MATH-500 and GSM8K, show LCPO reduces average output length by over 50% while preserving reasoning performance. The method requires only 0.8k training samples and 50 training steps, significantly lowering computational demands compared to prior approaches.

Key takeaway

For AI Engineers optimizing Large Reasoning Models for efficiency, LCPO offers a practical approach to significantly reduce output length without sacrificing performance. You should consider implementing LCPO to prune unnecessary CoT steps, especially when computational resources are constrained or when models exhibit overthinking. This method's low data and training step requirements (0.8k samples, 50 steps) make it highly adaptable for fine-tuning existing LRMs like Deepseek-R1-Distill-Qwen-7B on specific reasoning benchmarks.

Key insights

LCPO efficiently prunes lengthy LRM Chain-of-Thought outputs by balancing NLL loss with small-scale preference optimization.

Principles

• Shorter reasoning paths exist within LRM generation spaces.
• NLL loss can hinder length preference learning.
• Reward margins are crucial for length control.

Method

LCPO analyzes LRM generation paths, filters data by difficulty, and uses preference optimization to balance NLL-related implicit rewards, enabling efficient length control with minimal training data.

In practice

• Filter LRM outputs by question difficulty for concise training data.
• Use preference optimization to bias LRM trajectory distribution.
• Balance NLL loss with a counterpart term for length preference.

Topics

• Large Reasoning Models
• Chain-of-Thought Pruning
• Length Controlled Preference Optimization
• Preference Optimization
• Bradley-Terry Loss

Best for: AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• CLORE: Content-Level Optimization for Reasoning Efficiency — CLORE optimizes LLM reasoning efficiency by content-level editing of correct trajectories, reducing redundancy and illegibility.
• Beyond the Commitment Boundary: Probing Epiphenomenal Chain-of-Thought in Large Reasoning Models — Large reasoning models exhibit a "commitment boundary" where answers stabilize, allowing for significant CoT truncation.
• Token-weighted Direct Preference Optimization with Attention — Using an LLM's intrinsic attention to weight tokens during preference optimization significantly improves alignment and performance.
• Teaching Reasoning Models When to Stop: Data, Rewards, and Self-Aware Decoding — Efficient CoT reasoning requires balancing length reduction with accuracy, often by leveraging model confidence.
• SeLaR: Selective Latent Reasoning in Large Language Models — SeLaR improves LLM reasoning by selectively applying soft embeddings and regularization to enhance exploration at low-confidence steps.
• Free Energy Heuristics: Fast-And-Frugal Cognition as Active Inference Under Uncertain Precision — High meta-uncertainty causes CoT to degrade LLM accuracy by manufacturing false confidence.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.