Google's Gemma 4 AI models get 3x speed boost by predicting future tokens

2026-05-06 · Source: AI - Ars Technica · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Emerging Technologies & Innovation · Depth: Intermediate, short

Summary

Google has launched Multi-Token Prediction (MTP) drafters for its Gemma 4 open models, aiming to significantly accelerate local AI inference. Gemma 4, built on the same technology as Gemini AI but optimized for local execution, now operates under a more permissive Apache 2.0 license. MTP utilizes a form of speculative decoding where a lightweight drafter (e.g., 74 million parameters in Gemma 4 E2B) generates speculative tokens, which the main Gemma model then verifies in parallel. This process allows the system to produce multiple tokens in the time it previously took to generate one, effectively bypassing memory bandwidth limitations common in consumer hardware. Google reports MTP can make Gemma models up to three times faster, with specific gains like 2.8x and 3.1x on Pixel phones for E2B and E4B models, and a 2.5x boost for Gemma 4 31B on Apple M4 silicon, all without quality degradation.

Key takeaway

For NLP Engineers deploying Gemma 4 models on edge or consumer hardware, integrating Multi-Token Prediction (MTP) drafters is crucial for achieving substantial inference speedups. Your existing Gemma 4 deployments can see up to a 3x performance increase without compromising output quality, directly addressing common memory bandwidth bottlenecks. Explore the MTP-enabled models available through MLX, VLLM, SGLang, or Ollama to optimize your local AI applications and enhance user experience.

Key insights

Multi-Token Prediction (MTP) significantly accelerates local AI inference for Gemma 4 models via speculative decoding without quality loss.

Principles

• Speculative decoding improves token generation speed.
• Lightweight drafters can pre-generate tokens for verification.

Method

MTP uses a small drafter to generate speculative tokens, which the main model verifies in parallel, accepting sequences and generating an additional token simultaneously.

In practice

• Run Gemma 4 models up to 3x faster on consumer hardware.
• Improve battery life on mobile devices with E2B and E4B models.
• Utilize MTP drafters with MLX, VLLM, SGLang, or Ollama frameworks.

Topics

• Gemma 4
• Multi-Token Prediction
• Speculative Decoding
• Local AI Inference
• LLM Performance

Best for: NLP Engineer, Machine Learning Engineer, AI Engineer, AI Hardware Engineer

Related on AIssential

• Accelerating Gemma 4: faster inference with multi-token prediction drafters — Multi-Token Prediction drafters accelerate Gemma 4 inference up to 3x using speculative decoding without quality loss.
• MTP Layers for Gemma 4 and My Projects in Progress — Gemma 4's new MTP layers use speculative decoding to achieve up to 5x faster inference without accuracy loss.
• Gemma 4 MTP Local Test | Multi-Token Prediction of E2B using HuggingFace Transfomers | 🔴 Live — Gemma 4 models achieve 2-3x inference speedup via multi-token prediction using small drafter models and speculative decoding.
• Autoregressive Model Limits and Multi-Token Prediction in DeepSeek-V3 — Multi-Token Prediction (MTP) enhances language models by predicting multiple future tokens simultaneously during training.
• Google LiteRT-LM Speeds Up Local Inference Up to 2.2x With Gemma 4 Multi-Token Prediction — LiteRT-LM accelerates on-device LLM inference by up to 2.2x using MTP, memory locality, and optimized runtime for Gemma 4.
• Gemma 4 Is INCREDIBLE! Google's Open Model IS POWERFUL! (Fully Tested) — Gemma 4 models prioritize efficiency and agentic capabilities, enabling high performance on local and edge devices.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI - Ars Technica.