VIA-SD: Verification via Intra-Model Routing for Speculative Decoding

2026-06-10 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

VIA-SD (Verification via Intra-Model Routing for Speculative Decoding) introduces a multi-tier framework to enhance large language model inference efficiency by refining speculative decoding. Traditional speculative decoding uses a binary accept-or-recompute approach, often leading to full recomputation for moderately uncertain tokens. VIA-SD addresses this by employing a "slim submodel" derived from the full verifier via intra-model routing. This slim-verifier handles tokens requiring moderate verification resources, reducing expensive calls to the large model. The system processes draft tokens hierarchically: high-confidence tokens are directly accepted, medium-confidence tokens undergo slim-verifier regeneration, and only uncertain cases require full-model verification. Across four tasks and multiple model families, VIA-SD reduces rejection rates by 0.10-0.22 and achieves 10-20% speedups over strong speculative decoding baselines, alongside 2.5-3x acceleration compared to non-drafting decoding. It is also compatible with existing SD frameworks without requiring training modifications.

Key takeaway

For MLOps Engineers optimizing large language model inference, VIA-SD presents a compelling strategy to reduce operational costs and latency. You can achieve 10-20% speedups over current speculative decoding methods and 2.5-3x acceleration over non-drafting decoding. Implement this multi-tier approach, leveraging intra-model routing to create slim-verifiers, to process tokens hierarchically and minimize expensive full-model calls without modifying existing training procedures.

Key insights

Multi-tier speculative decoding with a slim-verifier significantly improves LLM inference efficiency by reducing full-model recomputations.

Principles

• Rejected tokens can be verified by slim submodels.
• Hierarchical verification reduces expensive model calls.
• Intra-model routing enables slim-verifier creation.

Method

VIA-SD processes draft tokens hierarchically: direct acceptance for high-confidence, slim-verifier regeneration for medium-confidence, and full-model verification for uncertain cases.

In practice

• Integrate multi-tier SD into existing frameworks.
• Route submodels from full verifiers for efficiency.
• Apply hierarchical token processing for LLM inference.

Topics

• Speculative Decoding
• LLM Inference
• Intra-Model Routing
• Multi-Tier Verification
• Model Efficiency

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• VIA-SD: Verification via Intra-Model Routing for Speculative Decoding — Speculative decoding efficiency improves significantly by introducing a multi-tier verification process with a routed slim-verifier.
• Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding — Hybrid Verified Decoding optimizes LLM inference by dynamically selecting between cache and model-based drafting based on predicted verification payoff.
• Speculative Decoding Scaling Laws (SDSL): Throughput Optimization Made Simple — SDSL analytically optimizes speculative decoding throughput by predicting optimal draft model sizes based on scaling laws.
• Your LLM Is Guessing Ahead. Then It Checks Itself aka Speculative Decoding — Speculative decoding accelerates LLM inference by using a small model to guess tokens, verified by a large model in one pass.
• WhiFlash: Accelerating Speculative Decoding with Token-Level Cross-Paradigm Routing — WhiFlash unifies autoregressive and diffusion-based speculative decoding via token-level routing to overcome static paradigm limitations.
• Teaching Diffusion to Speculate Left-to-Right — Aligning bidirectional diffusion drafters with left-to-right autoregressive verification significantly boosts speculative decoding efficiency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.