ThoughtFold: Folding Reasoning Chains via Introspective Preference Learning

2026-06-02 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

ThoughtFold is a novel framework designed to mitigate "over-thinking" issues in Large Reasoning Models (LRMs) that use Reinforcement Learning with Verifiable Rewards (RLVR) on Chain-of-Thoughts (CoTs). Existing RLVR methods often reinforce redundant explorations within long, outcome-correct CoT trajectories, leading to inefficient reasoning. ThoughtFold addresses this by employing fine-grained introspective preference learning. It identifies redundancy within individual correct trajectories to generate a spectrum of candidate sub-trajectories. A masked preference optimization objective then explicitly penalizes these redundant explorations, encouraging the model to form more concise reasoning paths. Experiments demonstrate that ThoughtFold significantly enhances efficiency, reducing the token usage of DeepSeek-R1-Distill-Qwen-7B by approximately 56% while maintaining accuracy.

Key takeaway

For Machine Learning Engineers optimizing Large Reasoning Models, ThoughtFold presents a critical advancement for improving inference efficiency. If your models exhibit "over-thinking" or high token usage with Chain-of-Thoughts, consider exploring introspective preference learning techniques. This approach can significantly reduce computational costs and latency, as demonstrated by a 56% token reduction on DeepSeek-R1-Distill-Qwen-7B, without sacrificing accuracy. Implementing similar fine-grained optimization could make your LRM deployments more practical and scalable.

Key insights

ThoughtFold uses introspective preference learning to prune redundant steps in reasoning chains, significantly boosting LRM efficiency.

Principles

• Redundant explorations in CoTs hinder LRM efficiency.
• Fine-grained preference learning can optimize reasoning paths.
• Introspection identifies sub-trajectories for conciseness.

Method

ThoughtFold applies introspective analysis to correct CoT trajectories, generating sub-trajectories. It then uses masked preference optimization to penalize redundant steps, folding reasoning chains into efficient paths.

In practice

• Reduce LRM inference costs by 56%.
• Optimize CoT generation for specific tasks.
• Improve reasoning efficiency on DeepSeek-R1-Distill-Qwen-7B.

Topics

• Large Reasoning Models
• Chain-of-Thought
• Preference Learning
• AI Efficiency
• Introspective Learning
• Reinforcement Learning

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Explaining Neural Networks in Preference Learning: a Post-hoc Inductive Logic Programming Approach — ILASP can approximate black-box Neural Networks for preference learning, even in high-dimensional spaces.
• Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability — Reasoning SFT can generalize cross-domain, but it is conditional on optimization, data quality, and model capability.
• CAPS: Cascaded Adaptive Pairwise Selection for Efficient Parallel Reasoning — CAPS optimizes LLM parallel reasoning by adaptively reducing verifier compute through cascaded pairwise selection.
• Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization — Large language models can learn and reproduce individual annotator explanation styles from human label variation.
• Think When Needed: Adaptive Reasoning-Driven Multimodal Embeddings with a Dual-LoRA Architecture — Adaptive reasoning in multimodal embeddings improves both efficiency and retrieval quality by selectively generating Chain-of-Thought.
• Efficient Exploration, Reasoning, and Training-Free MTP — Efficiency gains in LLMs are achievable through online RLHF, balanced reasoning control, and embedding-space multi-token prediction.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.