S2LC – 100 LoRA adapters in 3.59ms by reconstructing weights in GPU registers, never writing to HBM

2026-03-22 · Source: Machine Learning ML & Generative AI News · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

S2LC (Shared Spectral Low-Rank Compression) is a novel method that exploits shared spectral structure across neural network modules to efficiently serve numerous LoRA adapters. It computes a shared basis matrix V_common once per layer, encoding each module's unique contribution U_k into compact codebooks at approximately 3 bits per element. During inference, a fused Triton kernel reconstructs U_k values directly in GPU registers during tiled GEMM, eliminating intermediate HBM writes and achieving 10.1x memory compression over standard LoRA. This approach enables a 3.59 ms forward-pass latency for 100 concurrent adapters, with zero intermediate HBM writes verified by NVIDIA Nsight Compute. Future extensions are proposed for MoE expert compression, KV cache compression, and variable-depth serving.

Key takeaway

S2LC (Shared Spectral Low-Rank Compression) enables serving 100 LoRA adapters in just 3.59ms by reconstructing weights directly in GPU registers. This method achieves 10.1x memory compression over standard LoRA and eliminates intermediate HBM writes via a fused Triton kernel and CUDA Graph capture. It offers a significant performance boost for multi-adapter inference, with theoretical extensions to MoE and KV cache compression.

Topics

• LoRA Adapters
• Low-Rank Compression
• GPU Optimization
• Memory Compression
• Neural Network Inference

Code references

• QQQTech/S2LC

Best for: AI Engineer, NLP Engineer, MLOps Engineer, Machine Learning Engineer, Deep Learning Engineer, AI Architect

Related on AIssential

• H100 to B200: What Actually Changed — NVIDIA's Blackwell architecture, featuring the B200 GPU and GB200 NVL72 system, represents a fundamental shift from the H100, moving beyond individual GPU performance to datacenter-scale AI infrastructure.
• Convolutional Sparse Coding via the Locally Competitive Algorithm on Loihi 2 — Convolutional sparse coding via LCA is now implemented and benchmarked on Loihi 2, establishing a new benchmark for structured sparse inference on neuromorphic systems.
• Residual Modeling for High-Fidelity Learned Compression of Scientific Data — High-fidelity learned compression requires specialized residual coding to maintain efficiency, not generic global correction.
• From Naive to Near-Peak: Building High-Performance GEMM Kernels with Gluon — Achieving near-peak GPU kernel performance requires explicit hardware control and a profiling-driven, iterative optimization methodology.
• Behind Micron’s New 256GB LPDRAM: A Culture of Co-Design With Customers — Micron Technology has begun shipping customer samples of its 256GB SOCAMM2 LPDRAM module, touted as the industry's highest capacity LPDRAM for AI data center infrastructure, doubling the capacity of i…
• Unlocking the Edge deployment and ondevice acceleration of multi-LoRA enabled one-for-all foundational LLM — Efficient on-device LLM inference is achievable by integrating LoRAs, multi-stream decoding, and speculative decoding.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning ML & Generative AI News.