Adaptive Thinking: Large Language Models Know When to Think in Latent Space

2026-04-29 · Source: Apple Machine Learning Research · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Sonata (Self-Consistency-Guided Adapter for Thinking Allocation) is a novel, lightweight approach designed to optimize the performance-efficiency tradeoff in large language models (LLMs) by adaptively allocating thinking budgets. It leverages self-consistency, the agreement among multiple reasoning paths, as an indicator of a query's need for extended reasoning. Sonata operates by training an adapter offline on a calibration dataset to predict self-consistency from the last layer hidden representations during the query prefilling stage. This prediction then guides on-the-fly budget allocation before the LLM performs its chain-of-thought reasoning. The adapter is general, transferable across tasks, and introduces negligible computational overhead. Experiments with models like Qwen3-8B and GPT-OSS-120B on benchmarks such as GSM8K and MATH500 show Sonata reduces thinking tokens by 20% to 80% at equivalent accuracy, or improves accuracy by up to 5% at the same token cost.

Key takeaway

For NLP engineers optimizing LLM inference costs, Sonata offers a practical method to significantly reduce computational expenditure without sacrificing accuracy. By adaptively managing thinking budgets based on predicted self-consistency, you can achieve substantial token savings (20-80%) or accuracy gains (up to 5%). Consider integrating Sonata to enhance the efficiency of your CoT-enabled LLM deployments.

Key insights

Self-consistency can predict thinking necessity, enabling adaptive allocation of LLM reasoning budgets.

Principles

• Lower self-consistency indicates higher thinking necessity.
• Adaptive budget allocation optimizes performance-efficiency.

Method

Train an adapter offline to predict self-consistency from LLM hidden representations during prefilling, then use this prediction to guide on-the-fly thinking budget allocation.

In practice

• Integrate Sonata for 20-80% thinking token reduction.
• Apply Sonata to improve LLM accuracy by up to 5%.

Topics

• Large Language Models
• Chain-of-Thought Reasoning
• Self-Consistency
• Adaptive Thinking Allocation
• Sonata Adapter

Best for: NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Nvidia’s new technique cuts LLM reasoning costs by 8x without losing accuracy — Dynamic Memory Sparsification reduces LLM memory costs by intelligently compressing the KV cache without accuracy loss.
• Adaptive Interviewing for Persona Simulation in LLMs: Evidence-Grounded Reasoning Improves Decision Alignment — Adaptive interviewing improves LLM decision simulation by selectively grounding predictions in user-specific, follow-up derived evidence.
• Domain-Specialized Tree of Thought through Plug-and-Play Predictors — DST enhances LLM tree-based reasoning by using a lightweight, adaptive predictor for efficient, context-aware branch pruning.
• LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidance — LANG enhances multilingual LLM reasoning by using language-adaptive hints with a progressive decay schedule, preventing language drift.
• Adaptive Latent Agentic Reasoning — ALAR improves LLM agent efficiency by adaptively switching between compact latent reasoning and explicit chain-of-thought based on decision complexity.
• Self-Consistency from Only Two Samples: CoT-PoT Ensembling for Efficient LLM Reasoning — CoT-PoT ensembling improves LLM reasoning accuracy and efficiency by leveraging diverse, complementary reasoning modalities.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Apple Machine Learning Research.