AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention

2026-04-10 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

AsyncTLS is a hierarchical sparse attention system designed to enhance the efficiency of generative Large Language Model (LLM) inference for long contexts, addressing the challenges of quadratic attention complexity and prohibitive KV cache memory. It combines coarse-grained block filtering with fine-grained token selection to balance accuracy and efficiency. The system also features an asynchronous offloading engine that overlaps KV cache transfers with computation by exploiting temporal locality. Evaluated on models like Qwen3-8B, Qwen3-14B, and GLM-4.7-Flash across GQA and MLA architectures, AsyncTLS achieves accuracy comparable to full attention. It delivers significant performance improvements, including 1.2x-10.0x operator speedups and 1.3x-4.7x end-to-end throughput gains on context lengths ranging from 48k to 96k tokens, making it a scalable solution for ultra-long sequence generation.

Key takeaway

For AI Engineers optimizing LLM inference for ultra-long contexts, AsyncTLS offers a practical approach to mitigate memory and computational bottlenecks. You should consider implementing its two-level sparse attention and asynchronous KV cache offloading mechanisms to achieve substantial speedups and throughput improvements, particularly on GQA and MLA architectures. This method allows for accuracy comparable to full attention while significantly reducing resource demands, enabling larger batch processing and longer sequence lengths.

Key insights

AsyncTLS uses two-level sparse attention and asynchronous KV cache offloading for efficient long-context LLM inference.

Principles

• Combine coarse block filtering with fine token selection.
• Overlap KV cache transfers with computation.
• Exploit temporal locality for incremental data movement.

Method

AsyncTLS employs a two-level selection: coarse-grained block selection using a reformulated GEMM-friendly scoring, followed by fine-grained token selection within chosen blocks. It uses asynchronous prefetching and incremental block transmission for KV cache offloading.

In practice

• Implement hierarchical sparse attention for long contexts.
• Reformulate block scoring for GEMM efficiency.
• Utilize asynchronous prefetching for KV cache offloading.

Topics

• AsyncTLS
• Two-level Sparse Attention
• Asynchronous KV Cache Offloading
• Long-context LLM Inference
• Generative LLM Efficiency

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inference — ProxyKV uses an asynchronous small-model proxy to prune KV caches, achieving high accuracy and significant speedups for long-context LLM inference.
• Summarize Before You Speak with ARACH: A Training-Free Inference-Time Plug-In for Enhancing LLMs via Global Attention Reallocation — ARACH enhances LLMs by reallocating internal attention via a training-free context hub and logit offset at inference time.
• How To Reduce Inference Costs While Running LLMs — Optimizing LLM inference costs and performance requires a multi-faceted approach across hardware, software, and architectural techniques.
• Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion [R] — Orthrus accelerates LLM inference by integrating a diffusion module into a frozen AR Transformer, enabling parallel token generation.
• KV Caching in LLMs: A Guide for Developers — KV caching reuses attention's key and value projections to accelerate autoregressive LLM inference by avoiding redundant computation.
• Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention — LLM architectures are evolving with complex, targeted tweaks to optimize long-context efficiency and reduce computational overhead.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.