Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem

2026-07-10 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Machine unlearning in Large Language Models (LLMs) aims to remove specific knowledge while preserving other capabilities, crucial for privacy and safety. Current benchmarks, however, are unreliable, exhibiting "under-forgetting" where knowledge resurfaces via indirect queries, and "over-forgetting" where unrelated information is lost. These issues are identified as an asymmetric generalization problem, requiring forget evaluation to cover diverse query formulations and retain evaluation to probe a vast, implicitly defined set of disjoint facts. To address this, a new evaluation protocol and training corpus called SUITE has been introduced, which captures the forget-retain structure for real-world factual domains. Methods trained on SUITE show substantial improvement, underscoring the importance of training data alongside algorithmic design. Building on these insights, the JensUn++ unlearning algorithm was developed, demonstrating the best forget-retain utility trade-off across three LLMs in both sequential and joint unlearning scenarios. Code and datasets are publicly available.

Key takeaway

For Machine Learning Engineers developing unlearning capabilities in LLMs, you must move beyond simplistic benchmarks. Your evaluation strategies should incorporate diverse query formulations to prevent under-forgetting. Extensively probe unrelated knowledge to avoid over-forgetting. Consider integrating the SUITE protocol and training corpus. Explore JensUn++ to achieve superior forget-retain utility, as robust training data is crucial alongside algorithmic design for effective unlearning.

Key insights

Machine unlearning benchmarks fail due to asymmetric generalization, requiring robust evaluation and specialized training data for effective knowledge removal and retention.

Principles

• Forget evaluation needs diverse query formulations.
• Retain evaluation must probe all disjoint facts.
• Training data is critical for unlearning efficacy.

Method

SUITE provides an evaluation protocol and training corpus capturing forget-retain structure. JensUn++ is an unlearning algorithm achieving optimal forget-retain utility across LLMs in sequential and joint settings.

In practice

• Use SUITE for robust unlearning evaluation.
• Implement JensUn++ for better trade-offs.
• Prioritize training data in unlearning design.

Topics

• Machine Unlearning
• Large Language Models
• LLM Evaluation
• Asymmetric Generalization
• SUITE Protocol
• JensUn++ Algorithm

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance — Effective LLM unlearning requires implicitly identifying and selectively targeting forget-specific tokens based on their conflict with retaining general capabilities.
• Unlearning Under Imbalance: Benchmarking Fairness in Multimodal LLM Unlearning — Unbalanced unlearning requests in MLLMs can induce bias, necessitating benchmarks like FAIRGET and algorithms like FAUN to ensure fairness.
• Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance — LLM unlearning improves by learning token-level forget-specificity, defined by retain objective conflict, directly from model representations without external supervision.
• SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion — LLM unlearning can be retain-set-free by selectively demoting high-information tokens via self-distillation, preserving general utility.
• PreUnlearn: Auditing Collateral Knowledge Damage Before Large Language Model Unlearning — Collateral damage from LLM unlearning can be predicted pre-execution by analyzing data geometry, enabling proactive risk mitigation.
• TILDE: TILt-based Distributional Erasure for Concept Unlearning — Concept unlearning in diffusion models is best achieved by explicitly defining the desired post-unlearning distribution, not just suppressing concepts.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.