An MLIR-Based Compilation Method for Large Language Models

2026-07-20 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, long

Summary

Sophgo Inc. researchers present an MLIR-based compilation method for Large Language Models, addressing challenges in model import and efficient autoregressive inference on specialized AI accelerators. This method utilizes two custom dialects: TopOp, a high-level, framework-agnostic graph dialect for model semantics, and TpuOp, a target hardware dialect for chip-specific decisions like quantization and memory layout. A model is initially represented in TopOp, then lowered layer by layer to TpuOp, culminating in a deployable binary. Crucially, each Transformer layer is split into three static compilation stages—prefill, prefill_kv, and decode—to accommodate the distinct computational patterns of prompt-parallel processing and per-token generation. This approach is integrated into the TPU-MLIR compiler and LLM-TPU deployment project, supporting generative models such as Qwen, Llama, InternVL, and MiniCPM-V series, alongside quantization techniques like GPTQ, AWQ, and AutoRound.

Key takeaway

For AI Engineers deploying Large Language Models on specialized AI accelerators, adopting an MLIR-based compilation strategy with a three-stage Transformer layer split can significantly enhance efficiency. You should utilize framework-agnostic intermediate representations like TopOp for model import and target-specific dialects like TpuOp for hardware optimization. This approach enables static compilation for prompt processing (prefill, prefill_kv) and per-token generation (decode), reducing runtime overhead and improving performance on constrained hardware. Consider implementing these stages to optimize memory allocation and operator scheduling.

Key insights

MLIR-based compilation with a three-stage Transformer layer split optimizes LLM deployment on specialized AI accelerators.

Principles

• Compiler infrastructure should be reusable and extensible via custom dialects.
• Static compilation variants reduce dynamic overhead for autoregressive inference.
• High-level fused operators allow target-specific optimal fusion strategies.

Method

A trained LLM checkpoint is converted to framework-agnostic TopOp, then lowered to target-specific TpuOp, incorporating quantization and memory planning. Transformer layers are split into prefill, prefill_kv, and decode stages for static compilation.

In practice

• Use TopOp for hardware-agnostic model representation.
• Implement TpuOp for chip-specific optimizations.
• Split LLM inference into prefill, prefill_kv, decode.

Topics

• MLIR
• Large Language Models
• AI Accelerators
• Compiler Design
• Autoregressive Inference
• KV Cache Optimization

Code references

• sophgo/tpu-mlir
• sophgo/LLM-TPU

Best for: NLP Engineer, Computer Vision Engineer, AI Scientist, AI Engineer, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• An MLIR-Based Compilation Method for Large Language Models — MLIR-based compilation streamlines LLM deployment on accelerators by separating model semantics from hardware specifics and optimizing inference stages.
• Introducing Mellum2: A 12B Mixture-of-Experts Model by JetBrains — Mellum2 is a specialized, efficient MoE model for low-latency text and code workloads within larger AI systems.
• Zyphra Introduces Tensor and Sequence Parallelism (TSP): A Hardware-Aware Training and Inference Strategy That Delivers 2.6x Throughput Over Matched TP+SP Baselines — TSP simultaneously shards model weights and token sequences, resolving GPU memory bottlenecks for long-context transformers.
• Large Language Models vs Small Language Models — Small and large language models are distinct engineering responses to different constraints, often combined in hybrid systems.
• DeepSeek-V3: A Large-Scale MoE Pretraining Benchmark for MLPerf Training v6.0 — MLPerf Training v6.0 now benchmarks MoE LLMs, reflecting industry shifts to sparse computation and specialized architectures.
• Towards Faster Language Model Inference Using Mixture-of-Experts Flow Matching — MoE-FM enables faster, high-quality non-autoregressive language model inference by handling complex latent distributions.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.