Faster Gemma 4 on MLX with multi-token prediction

2026-06-28 · Source: Ollama Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Advanced, short

Summary

Ollama 0.31 significantly boosts Gemma 4's performance on Apple Silicon, achieving nearly 90% faster token generation on average across the Aider polyglot coding-agent benchmark. This speedup, enabled by default, comes from Multi-Token Prediction (MTP), where a small, fast draft model proposes several tokens that the main model then verifies in a single pass. This approach is particularly effective for predictable content like code, making coding agents more responsive. Ollama automatically tunes the optimal draft length at runtime, adapting to factors like model quantization, hardware, and text predictability to maximize tokens per second. Furthermore, a new MLX kernel contributed for batch verification accelerates Gemma 4's largest matrix multiplications by 2x to 2.5x on an M5 Max with nvfp4, by reusing weights across the batch. This improvement is currently available for Gemma 4 12B (nvfp4) in Ollama 0.31 for macOS.

Key takeaway

For AI engineers deploying Gemma 4 on Apple Silicon, especially for coding agent applications, you should immediately update to Ollama 0.31. This enables nearly 90% faster token generation via multi-token prediction and optimized MLX kernels, making your coding agents noticeably more responsive. Ensure you re-pull the "gemma4:12b-mlx" model to activate these performance improvements, gaining substantial throughput without altering model output.

Key insights

Multi-token prediction with auto-tuning and optimized batch verification dramatically accelerates LLM inference.

Principles

• Speculative decoding boosts LLM throughput.
• Auto-tuning draft length optimizes MTP.
• Batch verification kernels reduce redundant work.

Method

A draft model proposes token sequences, verified by the main model in a single pass. Ollama auto-tunes draft length based on acceptance rates and verification times, using rollback points for rejections.

In practice

• Use Ollama 0.31 for faster Gemma 4.
• Re-pull Gemma 4 12B (nvfp4) for MTP.
• Apply MTP for coding agent tasks.

Topics

• Gemma 4
• Multi-token Prediction
• Ollama
• MLX
• Speculative Decoding
• LLM Inference

Best for: Machine Learning Engineer, AI Engineer, AI Hardware Engineer

Related on AIssential

• Google's Gemma 4 AI models get 3x speed boost by predicting future tokens — Multi-Token Prediction (MTP) significantly accelerates local AI inference for Gemma 4 models via speculative decoding without quality loss.
• Gemma 4 + Ollama = FREE Claude Code Setup! — Gemma 4 models offer high intelligence per parameter, enabling efficient local AI inference across diverse hardware.
• Gemma 4 MTP Local Test | Multi-Token Prediction of E2B using HuggingFace Transfomers | 🔴 Live — Gemma 4 models achieve 2-3x inference speedup via multi-token prediction using small drafter models and speculative decoding.
• Accelerating Gemma 4: faster inference with multi-token prediction drafters — Multi-Token Prediction drafters accelerate Gemma 4 inference up to 3x using speculative decoding without quality loss.
• Google DeepMind’s Gemma 4: MoE, Efficiency Tricks, and Benchmarks — Gemma 4 offers a tiered family of multimodal models optimized for diverse hardware, balancing capability with deployment efficiency.
• Gemma 4 12B Is INCREDIBLE! BEST Local AI Coding Model! IS POWERFUL! (Fully Tested) — The Gemma 4 12B model offers competitive multimodal AI performance locally on consumer hardware via an efficient encoder-free architecture.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Ollama Blog.