JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models

2026-04-21 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, extended

Summary

Bitdefender researchers and the University of Bucharest have introduced JumpLoRA, a novel framework designed to enhance continual learning (CL) in Large Language Models (LLMs) by adaptively inducing sparsity in Low-Rank Adaptation (LoRA) blocks. This method leverages JumpReLU gating to achieve dynamic parameter isolation, which effectively mitigates catastrophic forgetting and task interference. JumpLoRA is highly modular and compatible with existing LoRA-based CL approaches, demonstrating significant performance improvements when integrated with IncLoRA and outperforming the state-of-the-art ELLA method on both the Standard CL Benchmark and the Long Sequence Benchmark. The framework works by learning a threshold alongside LoRA weights to cut off low-magnitude updates, creating sparse adapters that minimize overlap between tasks. Experiments were conducted on T5 models with 770M parameters using 8 Nvidia H200 GPUs, showing consistent gains in Overall Accuracy, Backward Transfer, and Forward Transfer across various task orders.

Key takeaway

For AI Engineers and Research Scientists working on continual learning for LLMs, JumpLoRA offers a robust solution to mitigate catastrophic forgetting. You should consider integrating JumpLoRA into your existing PEFT-based CL pipelines, especially if using IncLoRA or ELLA, as it consistently improves performance by adaptively inducing sparsity and reducing task interference. This approach allows for more efficient knowledge acquisition without costly full model retraining.

Key insights

JumpLoRA uses adaptive sparsity via JumpReLU gating to reduce catastrophic forgetting in LLM continual learning.

Principles

• Adaptive sparsity enhances parameter isolation.
• Dynamic thresholding can optimize weight updates.
• Modular frameworks improve existing CL methods.

Method

JumpLoRA applies a learnable JumpReLU function to LoRA weight updates, gradually introducing sparsification. It initializes a threshold based on parameter magnitudes and interpolates between dense and sparse updates, ensuring stable learning and effective parameter isolation.

In practice

• Integrate JumpLoRA with existing PEFT-based CL methods.
• Consider per-block sparsity thresholds for specific benchmarks.
• Less ELLA regularization may be needed with JumpLoRA.

Topics

• JumpLoRA
• Continual Learning
• Low-Rank Adaptation
• JumpReLU Gating
• Parameter Isolation

Code references

• huggingface/peft

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models — JumpLoRA uses JumpReLU gating to induce sparsity in LoRA blocks, preventing task interference in continual learning.
• KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptation — KeepLoRA++ balances continual learning objectives by adapting LoRA updates based on knowledge distribution across Transformer layers and subspaces.
• This AI Paper Introduces TinyLoRA, A 13-Parameter Fine-Tuning Method That Reaches 91.8 Percent GSM8K on Qwen2.5-7B — TinyLoRA enables high LLM performance with only 13 parameters by leveraging reinforcement learning.
• Learning, Fast and Slow: Towards LLMs That Adapt Continually [R] — FST combines slow parametric learning with fast in-context adaptation to enhance LLM performance and reduce forgetting.
• Revisiting Weight Regularization for Low-Rank Continual Learning — Weight regularization effectively mitigates task interference in low-rank continual learning with pre-trained models.
• ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decoding — ConfLayers uses confidence-based layer skipping to accelerate LLM inference without training a separate policy.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.