Sparser, Faster, Lighter Transformer Language Models

2026-05-09 · Source: Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Sakana AI, in collaboration with NVIDIA, introduced "Sparser, Faster, Lighter Transformer Language Models," a new ICML 2026 paper focusing on optimizing sparse transformer language models. The research addresses the paradox where unstructured sparsity in LLMs, despite reducing computation, often leads to slower execution due to irregular memory access on GPUs designed for dense operations. To resolve this hardware mismatch, the team developed a "Hybrid" format called TwELL (Tile-wise ELLPACK), which reshapes sparsity to better fit GPU architecture. This format dynamically routes 99% of highly sparse tokens through a fast path while using a dense backup matrix for rare, heavy tokens. The work also includes custom CUDA kernels that fuse multiple sparse matrix multiplications to maximize throughput and compress TwELL into a hybrid representation, minimizing activation sizes. Benchmarking sparse LLMs at billion-parameter scales demonstrated over 20% speedups and significant reductions in peak memory and energy consumption.

Key takeaway

For AI Engineers optimizing large language models, this research indicates that adapting sparsity to GPU architecture, rather than forcing GPUs to adapt to sparsity, is key for performance gains. You should explore integrating the open-source TwELL format and custom CUDA kernels into your sparse LLM training and inference pipelines to achieve over 20% speedups and substantial memory and energy savings.

Key insights

Reshaping LLM sparsity to fit GPU architecture significantly improves inference and training speed.

Principles

• Unstructured sparsity hinders GPU performance.
• Hardware-aware sparsity formats are crucial.
• Fuse sparse matmuls for higher throughput.

Method

The TwELL (Tile-wise ELLPACK) format dynamically routes sparse tokens through a fast path, backed by a dense matrix for heavy tokens. Custom CUDA kernels fuse sparse matmuls and compress TwELL for efficiency.

In practice

• Implement TwELL for sparse LLM inference.
• Utilize custom CUDA kernels for matmul fusion.
• Apply hybrid sparsity for memory savings.

Topics

• Transformer Language Models
• Sparse LLMs
• GPU Kernels
• TwELL (Tile-wise ELLPACK)
• CUDA

Code references

• SakanaAI/sparser-faster-llms

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Why I’m Learning GPU Programming for Faster AI Models — Efficient GPU memory management, not raw compute power, is key to accelerating AI model inference.
• Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression — Tensor decompositions for LLM compression face limits due to a mismatch with heterogeneous learned representations.
• I Built a C++ Backend So My GPU Would Stop Eating Air — Optimizing LLM inference requires VRAM-aware bin packing and efficient host-to-device data transfer to eliminate padding overhead.
• Prior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code Optimization — LLM agents in code optimization primarily leverage pretrained priors over iterative feedback or agentic exploration.
• Sakana AI and NVIDIA Introduce TwELL with CUDA Kernels for 20.5% Inference and 21.9% Training Speedup in LLMs — TwELL and new CUDA kernels exploit LLM sparsity for significant speedups and energy savings on NVIDIA GPUs.
• MIT Proved 90% of Every AI Model Is Dead Weight. It Took 8 Years for the Hardware to Catch Up. — Most neural network weights are redundant; efficient subnetworks can be trained directly with modern hardware and tooling.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Blog.