Knowledge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluation

2025-11-11 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Expert, extended

Summary

A new study introduces two metrics, Knowledge Separability Score (KSS) and Knowledge Persistence Score (KPS), to evaluate Multilingual Machine Unlearning (MMU) in Large Language Models (LLMs). Existing MMU evaluations, which are often direct extensions of per-language protocols, fail to adequately capture the cross-linguistic distribution of sensitive information. The researchers developed a multilingual parallel QA dataset of 3,800 instances across 10 languages, including high- and low-resource languages, to simulate direct memorization and indirect cross-linguistic spread. KSS measures overall unlearning quality across multiple languages, while KPS assesses consistent information removal between different language pairs. Experiments on Llama3.1-8B-Instruct and Qwen3-4B-Instruct using various unlearning methods, including Gradient Ascent (GA), Gradient Ascent with Gradient Descent term (GAGDR), Gradient Ascent with KL minimization (GAKLR), Negative Preference Optimization (NPO), and pruning, revealed that unlearning is more challenging in hold-out languages and performance degrades as the forget ratio increases.

Key takeaway

For NLP engineers developing or deploying multilingual LLMs, current unlearning evaluation methods are insufficient for ensuring privacy across languages. You should integrate the Knowledge Separability Score (KSS) and Knowledge Persistence Score (KPS) into your evaluation pipelines to accurately assess whether sensitive information has been truly removed across all relevant languages, including those not directly used in training. This will help mitigate privacy risks from cross-linguistic knowledge persistence.

Key insights

New metrics improve Multilingual Machine Unlearning evaluation by accounting for cross-linguistic knowledge spread.

Principles

• Knowledge in MMU is distributed across languages.
• Cross-linguistic spread of knowledge can persist post-unlearning.
• Unlearning performance degrades with increased forget ratio.

Method

The study generated a 10-language, 3,800-instance multilingual parallel QA dataset from synthetic profiles, then applied KSS and KPS to evaluate unlearning methods on Llama3.1-8B-Instruct and Qwen3-4B-Instruct.

In practice

• Use KSS-ROC and KSS-PR for comprehensive unlearning quality.
• Employ KPS to quantify cross-lingual knowledge persistence.
• Consider hold-out languages for robust MMU evaluation.

Topics

• Multilingual Machine Unlearning
• Knowledge Separability Score
• Knowledge Persistence Score
• LLM Privacy Risks
• Cross-linguistic Knowledge Spread

Code references

• joke2k/faker

Best for: NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• Knowledge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluation — New metrics evaluate how effectively multilingual LLMs remove information across different languages.
• PreUnlearn: Auditing Collateral Knowledge Damage Before Large Language Model Unlearning — Collateral damage from LLM unlearning can be predicted pre-execution by analyzing data geometry, enabling proactive risk mitigation.
• Detecting Sensitive Personal Information in Japanese Pre-Training Corpora for Large Language Models — This study introduces the first effective method for detecting special care-required personal information (SCPI) in Japanese LLM corpora.
• Culturally-Adapted Red-Teaming Across East and Southeast Asian Contexts: A Methodological and Comparative Analysis — Direct translation of safety benchmarks fails to capture cultural context, underestimating LLM risks in multilingual settings.
• Building Trust in the Skies: A Knowledge-Grounded LLM-based Framework for Aviation Safety — Combining LLMs with dynamically updated Knowledge Graphs enhances trustworthiness and verifiability in safety-critical aviation analytics.
• Rethinking Cross-lingual Gaps from a Statistical Viewpoint — LLM cross-lingual gaps stem from response variance, not knowledge barriers, and are reducible via variance control.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.