Dictionary-Based Speculative Decoding for Non-Latin-Script Languages

2026-06-28 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

Dictionary-Based Speculative Decoding (DictSpec) is proposed to address inefficient tokenization in large language models for non-Latin-script languages like Ukrainian and Crimean Tatar, where words can split into 2-3 times more tokens than English. DictSpec accelerates inference by using a static n-gram lookup table, built offline from an unlabeled corpus, to propose draft continuations. This table requires no trainable parameters or GPU resources, is inexpensive to construct, adds under 5 MB memory overhead, and is reusable across models sharing a tokenizer. Evaluated on 3B to 70B parameter models via a vLLM plugin, DictSpec reduced verification steps by up to 1.65x in emulation. A hybrid approach with prompt-local n-gram speculation achieved up to 1.76x speedup in live vLLM serving. The code and vLLM plugin are open-source.

Key takeaway

For Machine Learning Engineers optimizing LLM inference for non-Latin languages, DictSpec offers a resource-efficient method to significantly improve throughput. You should consider integrating this dictionary-based speculative decoding approach, especially for languages with high tokenizer fertility, to reduce verification steps and achieve substantial speedups without additional GPU investment. Explore the open-source vLLM plugin to evaluate its impact on your specific models and datasets.

Key insights

DictSpec accelerates non-Latin-script LLM inference by using an offline n-gram lookup table to propose draft continuations.

Principles

• Inefficient tokenization impacts non-Latin scripts.
• Static n-gram tables can accelerate LLM inference.
• Tokenizer fertility correlates with inference gains.

Method

DictSpec builds a static n-gram lookup table offline from an unlabeled corpus. This table proposes draft continuations, reducing verification steps during LLM inference without requiring GPU resources or trainable parameters.

In practice

• Implement DictSpec as a vLLM plugin.
• Build n-gram tables for specific non-Latin languages.
• Combine DictSpec with prompt-local n-gram speculation.

Topics

• Speculative Decoding
• LLM Inference
• Non-Latin Scripts
• Tokenization
• vLLM
• N-gram Models

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• The Next Frontier: How Speculative Decoding Is Eating the LLM Inference Stack — Speculative decoding boosts LLM inference speed by parallelizing token verification, leveraging idle GPU compute without quality loss.
• Making LLMs faster and more efficient across multiple languages — ADASPEC enables efficient multilingual LLM inference by dynamically adapting language-specific drafters and vocabularies through self-synthesized data.
• AdaPLD: Adaptive Retrieval and Reuse for Efficient Model-Free Speculative Decoding — AdaPLD improves model-free speculative decoding by adaptively combining lexical and semantic retrieval with branched draft hypotheses.
• DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification — Dynamic ensemble verification in speculative decoding significantly boosts token acceptance and inference speed without sacrificing quality.
• DFlash vs MTP: Qwen3.6 Speculative Decoding Benchmarks with vLLM and llama.cpp — Speculative decoding via DFlash or MTP can significantly accelerate LLM inference if configured correctly.
• FairyFuse: Multiplication-Free LLM Inference on CPUs via Fused Ternary Kernels — Ternary weights enable multiplication-free LLM inference on CPUs, significantly boosting throughput while preserving quality.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.