ConFu: Contemplate the Future for Better Speculative Sampling

2026-03-11 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, extended

Summary

ConFu (Contemplate the Future) is a novel speculative decoding framework designed to accelerate large language model (LLM) inference by improving draft model quality. It addresses the limitation of existing draft models, like the EAGLE series, which suffer from error accumulation by only conditioning on the current prefix. ConFu enables draft models to anticipate the future direction of generation through three innovations: (i) "contemplate tokens" and "soft prompts" that provide future-oriented signals from the target model at negligible cost, (ii) a dynamic contemplate token mechanism using Mixture-of-Experts (MoE) for context-aware future prediction, and (iii) a training framework with "anchor token sampling" and "future prediction replication" for robust learning. Experiments on SpecBench with Llama-3 3B and 8B models demonstrate that ConFu improves token acceptance rates and generation speed over EAGLE-3 by 8-11% across various downstream tasks, sampling temperatures, and draft tree budgets.

Key takeaway

For NLP Engineers optimizing LLM inference, ConFu offers a significant advancement in speculative decoding. By enabling draft models to anticipate future generation, it consistently improves token acceptance rates and decoding speed by 8-11% over EAGLE-3. You should consider integrating ConFu's future-aware signals and dynamic contemplate tokens to mitigate error accumulation and achieve higher throughput in your LLM deployments, especially for Llama-3 3B and 8B models.

Key insights

ConFu enhances speculative decoding by enabling draft models to anticipate future generation direction via target model "thought" signals.

Principles

• Future-aware signals improve draft model accuracy.
• Dynamic tokens adapt to diverse generation contexts.
• Robust training enhances future prediction stability.

Method

ConFu uses contemplate tokens and soft prompts to extract future-oriented signals from the target model. A dynamic MoE mechanism adapts these tokens, and a training framework with anchor token sampling and future prediction replication ensures robust learning.

In practice

• Integrate contemplate tokens for future prediction.
• Employ MoE for dynamic, context-aware signals.
• Use anchor token sampling to reduce training memory.

Topics

• Speculative Decoding
• Large Language Models
• Inference Acceleration
• Draft Models
• Mixture-of-Experts

Code references

• SafeAILab/EAGLE

Best for: NLP Engineer, AI Scientist, Research Scientist, AI Researcher, AI Engineer, Machine Learning Engineer

Related on AIssential

• Your LLM Is Guessing Ahead. Then It Checks Itself aka Speculative Decoding — Speculative decoding accelerates LLM inference by using a small model to guess tokens, verified by a large model in one pass.
• Why MoE models get more from speculative decoding - Cohere — MoE sparsity and expert routing significantly enhance speculative decoding performance, especially at moderate batch sizes.
• JetFlow: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting — JetFlow breaks speculative decoding's scaling ceiling by combining one-forward drafting efficiency with branch-wise causal conditioning.
• Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing — Window-level optimization via reinforcement learning significantly improves speculative decoding efficiency for LLMs.
• DFlash for Qwen3.5, EAGLE for Gemma 4, and the MiniMax M2.7 License Debate — Speculative decoding significantly accelerates LLM inference by using a draft model to propose tokens for a more powerful target model to verify.
• Large model inference container – latest capabilities and performance enhancements — LMCache and EAGLE speculative decoding significantly reduce LLM inference costs and latency by reusing KV caches and accelerating token generation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.