Unlearning Under Imbalance: Benchmarking Fairness in Multimodal LLM Unlearning

2026-07-23 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

A new Visual Question Answering (VQA) benchmark, FAIRGET, has been introduced to evaluate machine unlearning in Multimodal Large Language Models (MLLMs) under realistic, unbalanced forget requests. Unlike prior benchmarks that assume uniformly distributed unlearning requests, FAIRGET simulates scenarios where different demographic groups request unlearning at varying frequencies, which can lead to biased model behaviors. This benchmark addresses a critical gap in current evaluation methods for MLLMs. Additionally, the paper proposes FAUN, the first unlearning algorithm specifically designed for MLLMs that not only forgets unlearning data but also preserves model fairness. FAUN achieves this by employing a bias-aware activation steering mechanism to account for the unbalanced nature of the forget data. Experiments conducted on both FAIRGET and the established FIUBench demonstrate FAUN's superior performance in terms of both unlearning quality and fairness.

Key takeaway

For Machine Learning Engineers developing or deploying Multimodal LLMs, you must account for the fairness implications of unlearning requests. If you are implementing unlearning mechanisms, consider that real-world forget requests are often unbalanced, potentially introducing bias. You should utilize benchmarks like FAIRGET to rigorously evaluate fairness in your unlearning solutions. Explore algorithms such as FAUN, which specifically addresses bias through activation steering, to ensure your models remain fair post-unlearning.

Key insights

Unbalanced unlearning requests in MLLMs can induce bias, necessitating benchmarks like FAIRGET and algorithms like FAUN to ensure fairness.

Principles

• Real-world unlearning requests are often unbalanced.
• Unbalanced unlearning can create model bias.
• Fairness is crucial in MLLM unlearning.

Method

FAUN, an MLLM unlearning algorithm, employs a bias-aware activation steering mechanism. This method specifically accounts for the unbalanced nature of forget data to unlearn identities while preserving model fairness.

In practice

• Benchmark MLLM unlearning with FAIRGET.
• Implement FAUN for fair MLLM unlearning.

Topics

• Machine Unlearning
• Multimodal LLMs
• Fairness Benchmarking
• Bias Mitigation
• Visual Question Answering
• FAUN Algorithm

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, Machine Learning Engineer, AI Ethicist

Related on AIssential

• Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks — Multimodal unlearning selectively removes sensitive data from foundation models while preserving utility.
• Bring Your Own Prompts: Use-Case-Specific Bias and Fairness Evaluation for LLMs — LLM bias and fairness evaluation requires a use-case-specific approach, as risks vary significantly across deployment contexts.
• Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem — Machine unlearning benchmarks fail due to asymmetric generalization, requiring robust evaluation and specialized training data for effective knowledge removal and retention.
• Fair Cognitive Impairment Detection Through Unlearning — Demographic unlearning via gradient reversal in multimodal fusion improves fair and robust MCI detection from speech.
• QuantiBias: Benchmarking Quantization-Induced Bias in LLMs — LLM quantization measurably increases bias in open-ended generation, a critical side effect missed by standard safety evaluations.
• Evaluating and Mitigating Gender Bias in Pre-trained Embeddings for ML-based Recruitment — Gender bias persists in ML recruitment embeddings even after scrubbing, requiring multi-faceted mitigation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.