Unlocking Sparse Acceleration on AMD GPUs with hipSPARSELt

2026-02-17 · Source: AMD ROCm Blogs · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, medium

Summary

AMD has released hipSPARSELt, a high-performance library designed to accelerate sparse matrix operations on AMD GPUs, particularly for AI models like LLaMA and DINOv2 ViT-L. The library leverages the 2:4 structured sparsity pattern, where every group of four consecutive weights contains exactly two zeros, enabling hardware-friendly optimizations and efficient compression. This pattern yields compression ratios of 56.25% for Float16/BFloat16 and 62.5% for Int8/Float8 data types. hipSPARSELt supports 2:4 structured sparsity matrix multiplication accelerated by AMD Matrix Core instructions, provides pruning and compression utilities, and offers operation fusion for activations, scalar multipliers, and bias vectors. Benchmarks on an AMD MI300 GPU using FP16 precision demonstrated significant speedups, approximately 1.3x, compared to dense GEMM operations performed with hipBLASLt.

Key takeaway

For AI Engineers and Machine Learning Engineers optimizing large models on AMD hardware, hipSPARSELt offers a direct path to significant performance gains. By adopting 2:4 structured sparsity and integrating this library, you can achieve up to 1.3x speedup in matrix multiplication, reducing memory footprint and accelerating inference for models like LLaMA. Consider migrating existing dense GEMM operations to hipSPARSELt for improved efficiency.

Key insights

hipSPARSELt accelerates sparse matrix operations on AMD GPUs using 2:4 structured sparsity for improved AI model efficiency.

Principles

• 2:4 sparsity prunes two elements from every four.
• Structured sparsity enables hardware optimization.

Method

The process involves pruning a dense matrix to 2:4 sparsity by zeroing out two elements per group of four, then compressing the remaining values and their indices for efficient storage and computation.

In practice

• Use hipSPARSELt for sparse AI model deployment.
• Target AMD MI300 series GPUs for optimal performance.

Topics

• hipSPARSELt
• Structured Sparsity
• AMD GPUs
• Sparse Matrix Multiplication
• ROCm

Code references

• ROCm/rocm-libraries

Best for: AI Engineer, Machine Learning Engineer, Software Engineer

Related on AIssential

• Sparser, Faster, Lighter Transformer Language Models — Reshaping LLM sparsity to fit GPU architecture significantly improves inference and training speed.
• Building and Deploying Custom hipBLASLt Libraries on AMD Instinct GPUs — Custom "hipBLASLt" builds offer precise GPU optimization and flexible deployment for AMD Instinct GPUs, avoiding system-wide conflicts.
• Introducing hipThreads: A C++ - Style Concurrency Library for AMD GPUs — hipThreads enables incremental GPU acceleration of C++ `std::thread` code using familiar concurrency patterns and persistent multi-fiber execution.
• Convolutional Sparse Coding via the Locally Competitive Algorithm on Loihi 2 — Convolutional sparse coding via LCA is now implemented and benchmarked on Loihi 2, establishing a new benchmark for structured sparse inference on neuromorphic systems.
• SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference — SharQ combines activation sparsity and FP4 quantization for LLM inference via an online sparse-dense decomposition.
• Randomized Subspace Nesterov Accelerated Gradient — RS-NAG methods accelerate optimization by using low-dimensional projected gradients, outperforming standard Nesterov acceleration in specific oracle complexity scenarios.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AMD ROCm Blogs.