Ragged Paged Attention: A High-Performance and Flexible LLM Inference Kernel for TPU

2026-04-16 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Expert, quick

Summary

Ragged Paged Attention (RPA) is a high-performance and flexible attention kernel designed for Large Language Model (LLM) inference on Google's Tensor Processing Units (TPUs). Developed using Pallas and Mosaic, RPA addresses the challenges of dynamic and ragged execution patterns prevalent in modern LLM serving, which are not well-supported by existing GPU-centric kernels. RPA employs three core techniques: fine-grained tiling for efficient dynamic slicing over ragged memory, a custom software pipeline that fuses KV cache updates with attention computation, and a distribution-aware compilation strategy that creates specialized kernels for decode, prefill, and mixed workloads. Evaluated on Llama 3 8B on TPU7x, RPA achieves up to 86% memory bandwidth utilization (MBU) in decode and 73% model FLOPs utilization (MFU) in prefill. It is integrated into vLLM and SGLang as a production-grade TPU backend.

Key takeaway

For AI Architects and MLOps Engineers deploying LLMs on TPUs, RPA offers a critical solution for optimizing inference performance and TCO. Its specialized kernel design for dynamic workloads, including fused KV cache updates and distribution-aware compilation, directly addresses the limitations of GPU-centric approaches. Consider adopting RPA within vLLM or SGLang to significantly improve memory bandwidth and FLOPs utilization for your TPU-based LLM deployments, especially with models like Llama 3 8B.

Key insights

Ragged Paged Attention optimizes LLM inference on TPUs for dynamic workloads via fine-grained tiling and fused operations.

Principles

• Fuse KV cache updates with attention.
• Specialize kernels for workload types.
• Utilize fine-grained tiling for ragged memory.

Method

RPA uses fine-grained tiling for dynamic slicing, a software pipeline fusing KV cache updates with attention, and distribution-aware compilation to generate specialized kernels for decode, prefill, and mixed LLM workloads on TPUs.

In practice

• Integrate RPA for Llama 3 8B on TPU7x.
• Use Pallas and Mosaic for kernel development.
• Optimize for ragged memory patterns.

Topics

• Ragged Paged Attention
• LLM Inference
• Tensor Processing Units
• KV Cache
• Pallas and Mosaic

Best for: AI Architect, MLOps Engineer, CTO, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Ragged Paged Attention: A High-Performance and Flexible LLM Inference Kernel for TPU — RPA optimizes LLM inference on TPUs by handling dynamic workloads and fusing KV cache updates for high utilization.
• Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention — LLM architectures are evolving with complex, targeted tweaks to optimize long-context efficiency and reduce computational overhead.
• llama.cpp: Fast Local LLM Inference, Hardware Choices & Tuning — llama.cpp is the dominant open-source C/C++ framework enabling fast local large language model (LLM) inference, leveraging hardware advancements like NVIDIA's RTX 5090 and Apple's M4 Ultra for privacy…
• Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving — Head-wise KV cache retention patterns are stable and model-intrinsic, enabling deterministic optimization for non-uniform compression.
• LLM Inference and Optimization: Fundamentals, Bottlenecks, and Techniques — Optimizing LLM inference requires understanding distinct prefill and decode phases, leveraging KV caching, and applying advanced techniques.
• Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving — SPIN unifies sparse attention and hierarchical memory to significantly boost long-context LLM serving efficiency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.