MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMs

2026-06-11 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, quick

Summary

MLUBench is introduced as a large-scale, comprehensive benchmark designed to evaluate lifelong unlearning in Multimodal Large Language Models (MLLMs). This benchmark features 127 entities across 9 classes, addressing the challenge of sequential data unlearning requests. Experiments using MLUBench reveal that current unlearning methods suffer from severe, cumulative degradation. A critical finding is that MLLM lifelong unlearning uniquely requires preserving multimodal alignment; unlearning from one modality can degrade the entire model. To counter this, the paper proposes LUMoE, an effective method that significantly mitigates the observed degradation. The MLUBench dataset and source code are open-sourced.

Key takeaway

For machine learning engineers developing or deploying MLLMs, recognize that data unlearning requests present a unique challenge due to the need to preserve multimodal alignment. Existing methods often lead to cumulative model degradation. You should evaluate your unlearning strategies against the MLUBench dataset and explore methods like LUMoE to ensure robust model performance while complying with data removal requests.

Key insights

MLLM lifelong unlearning uniquely struggles with multimodal alignment degradation, which MLUBench and LUMoE address.

Principles

• MLLM lifelong unlearning must preserve multimodal alignment.
• Unlearning from one modality can degrade the entire MLLM.
• Existing unlearning methods show severe, cumulative degradation.

Method

LUMoE is proposed as an effective method to mitigate cumulative degradation in MLLM lifelong unlearning, specifically addressing multimodal alignment preservation.

In practice

• Utilize MLUBench for MLLM lifelong unlearning evaluation.
• Consider LUMoE to mitigate MLLM unlearning degradation.

Topics

• Multimodal LLMs
• Lifelong Unlearning
• Data Unlearning
• MLUBench
• Multimodal Alignment
• LUMoE

Code references

• lihe-maxsize/Lifelong_Unlearning_main

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMs — MLLM lifelong unlearning uniquely challenges multimodal alignment, requiring isolated, modular solutions to prevent cumulative degradation.
• MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models — MEMLENS benchmarks multimodal long-term memory, revealing distinct strengths and weaknesses in LVLMs and memory-augmented agents.
• Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpages — WebIGBench is the first benchmark to evaluate MLLMs on interactive webpage code generation, addressing gaps in existing static-focused evaluations.
• Detoxification for LLM: From Dataset Itself — Detoxifying LLM pretraining datasets directly at the source fundamentally reduces inherent model toxicity while preserving semantics.
• ATLAS: All-round Testing of Long-context Abilities across Scales — Long-context LLM evaluation requires length-dependent capability profiling to reveal performance degradation and task-specific strengths.
• VCIFBench: Evaluating Complex Instruction Following for Video Understanding — MLLMs struggle with complex, constraint-rich video instruction following, highlighting a critical evaluation gap.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.