Serving MiniMax-M3 for efficient inference: Unlocking 1M-Token Context and Multimodality Without Regrets

2026-06-10 · Source: Together AI | The AI Native Cloud - Together.ai · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Software Development & Engineering · Depth: Expert, medium

Summary

Together AI has become the preferred cloud partner for MiniMax M3, a new multimodal model featuring a 1M-token context window and native multimodal reasoning. Upon its open-weights release, Together AI will host M3 as a developer endpoint. Together AI's Inference and Kernel teams implemented significant engineering breakthroughs to efficiently serve M3, achieving 81-125% throughput improvements across various concurrency levels. Key optimizations include a KV-Block-Major sparse attention kernel, a novel paged attention integration for MiniMax Sparse Attention (MSA), an optimized decode index scoring kernel, and a Rust-based multimodal preprocessing gateway (SMG). MSA, a core architectural change, reduces attention computation cost by capping tokens each query attends to, leading to over 9x speedup in prefilling and 15x in decoding. The SMG handles all vision preprocessing on the CPU, freeing GPU resources.

Key takeaway

For AI Engineers deploying large multimodal models with extensive context windows, you should prioritize deep systems-level optimizations. Consider implementing sparse attention kernels and offloading multimodal preprocessing to a CPU-based gateway like SMG. This approach can yield substantial throughput gains, as demonstrated by Together AI's 81-125% improvements for MiniMax M3, making 1M-token context and multimodality economically viable for production. Evaluate your inference stack for similar kernel and preprocessing opportunities.

Key insights

Efficiently serving large multimodal models with long contexts requires deep systems-level optimization across multiple architectural components.

Principles

• Sparse attention significantly reduces long-context computation.
• Offload multimodal preprocessing to CPU gateways.
• Optimize kernel execution for specific attention architectures.

Method

Together AI's method involved developing a KV-Block-Major sparse attention kernel, integrating MSA with paged attention, optimizing decode index scoring, and implementing a Rust-based multimodal preprocessing gateway.

In practice

• Use KV-Block-Major for sparse attention.
• Implement Rust gateway for multimodal preprocessing.
• Adapt paged attention for sparse models.

Topics

• MiniMax M3
• Multimodal AI
• Sparse Attention
• Inference Optimization
• Paged Attention
• Together AI

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• DeepSeek-V4 Review: Why Million-Token Context Needs Efficient Attention, Not Just Larger Windows — Efficient attention mechanisms and specialized training are key to practical 1M-token context LLMs.
• MiniMax Releases MiniMax M3 with MSA Architecture Supporting 1M-Token Context, Native Multimodality, and Agentic Coding — MiniMax M3's MSA architecture dramatically improves long-context efficiency and enables native multimodality and advanced agentic coding.
• Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads — Multimodal LLMs exhibit functional sparsity, relying on a sparse subset of "CoRe heads" for efficient, precise cross-modal visual information retrieval.
• How To Reduce Inference Costs While Running LLMs — Optimizing LLM inference costs and performance requires a multi-faceted approach across hardware, software, and architectural techniques.
• How Much Dense Attention is Necessary? Oracle-Guided Sparse Prefill for Full/GQA Layers in Hybrid Long-Context Models — The study quantifies dense attention needs for long-context models, achieving significant speedups with minimal quality loss via sparse prefill.
• Nanbeige4.1: Only 3B Parameters, but as Good as Qwen3 32B? — Advanced post-training and hybrid attention mechanisms enable compact and massive LLMs to achieve impressive performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Together AI | The AI Native Cloud - Together.ai.