Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Models

2026-05-14 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Octopus is a novel two-stage continual learning framework designed for multimodal large language models (MLLMs) that addresses catastrophic forgetting without relying on historical task data. This framework, based on History-Free Gradient Orthogonalization (HiFGO), enforces gradient-level orthogonality to prevent parameter interference. Unlike architecture-based methods that add computational overhead or rehearsal-based methods with privacy and storage concerns, Octopus decouples task adaptation from regularization. This strategy aims to balance model plasticity and stability during sequential knowledge acquisition. Experimental results on the UCIT benchmark demonstrate that Octopus achieves state-of-the-art performance, outperforming previous methods by 2.14% in average accuracy and 6.82% in last task accuracy.

Key takeaway

For AI Engineers developing continual learning systems for MLLMs, Octopus offers a compelling alternative to data-intensive or computationally heavy methods. By adopting its history-free gradient orthogonalization, you can mitigate catastrophic forgetting while addressing privacy and storage concerns. Consider implementing this two-stage finetuning approach to achieve a better balance between model plasticity and stability in your sequential learning tasks.

Key insights

Octopus enables continual learning in MLLMs by orthogonalizing gradients without needing historical data.

Principles

• Decouple task adaptation from regularization.
• Enforce gradient orthogonality for stability.

Method

Octopus employs a two-stage finetuning strategy using History-Free Gradient Orthogonalization (HiFGO) to achieve gradient-level orthogonality, balancing plasticity and stability in MLLMs.

In practice

• Apply HiFGO for privacy-sensitive continual learning.
• Use two-stage finetuning for MLLM stability.

Topics

• Continual Learning
• Multimodal Large Language Models
• Gradient Orthogonalization
• Catastrophic Forgetting
• Finetuning Strategy

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Mitigating Forgetting in Continual Learning with Selective Gradient Projection — SFAO uses similarity-gated gradient updates to balance plasticity and stability in continual learning, reducing forgetting and memory.
• Continual Fine-Tuning with Provably Accurate and Parameter-Free Task Retrieval — Proteus offers provably accurate, parameter-free task retrieval for continual fine-tuning by leveraging distinct representation signatures.
• ECA: Efficient Continual Alignment for Open-Ended Image-to-Text Generation — ECA enables efficient continual alignment in OpenITG by adapting VLM alignment modules to evolving visual data without prior raw data.
• Fast and Slow Variational Continual Learning — Continual IVON (CoVON) integrates fast and slow adaptation via posterior merging within VCL, improving stability and plasticity in deep networks.
• Self-orthogonalizing attractor neural networks emerging from the free energy principle — Attractor networks emerge from the Free Energy Principle, yielding self-organizing, orthogonal representations and robust learning.
• Similarity-Aware Mixture-of-Experts for Data-Efficient Continual Learning — An adaptive MoE framework improves continual learning with scarce, overlapping data via similarity awareness.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.