Membership Inference Attacks for Unseen Classes

2025-06-06 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Expert, short

Summary

The paper "Membership Inference Attacks for Unseen Classes" by Thaker et al., published at ICML 2026, addresses a critical limitation in data auditing for AI models. Current membership inference attacks (MIAs) typically require access to representative samples of harmful content from the target distribution, which is often legally or ethically restricted for real-world auditors. The authors formalize this as the "unseen class" setting, demonstrating that state-of-the-art MIAs fail under these conditions. They show that quantile regression attacks significantly outperform traditional shadow model-based approaches, achieving up to 11x the True Positive Rate in practice. This work highlights a crucial failure mode in existing MIA tools, urging caution for practitioners in AI safety applications.

Key takeaway

For AI Security Engineers tasked with auditing black-box models for harmful training data, you should re-evaluate the efficacy of your current membership inference attack (MIA) tools. Existing state-of-the-art MIAs are likely to fail in real-world "unseen class" scenarios where representative harmful content is inaccessible. Prioritize implementing quantile regression attacks, which demonstrate up to 11x higher True Positive Rates, to ensure more effective and ethically compliant data auditing practices.

Key insights

Existing MIAs fail when auditors lack representative "unseen class" data, making quantile regression attacks superior for real-world auditing.

Principles

• Data auditing requires robust MIAs.
• Access to full target distribution is critical.
• Generalization properties are key for success.

Method

The paper formalizes the "unseen class" data access model and empirically/theoretically demonstrates that quantile regression attacks achieve superior True Positive Rates compared to shadow model-based MIAs in this setting.

In practice

• Adopt quantile regression for MIAs.
• Re-evaluate MIA tool efficacy.

Topics

• Membership Inference Attacks
• AI Safety
• Data Auditing
• Quantile Regression
• Black-box Models
• Unseen Class Setting

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, AI Security Engineer, AI Ethicist

Related on AIssential

• Detecting Data Contamination in Large Language Models — Black-box MIAs are currently ineffective at reliably detecting data contamination in LLM training corpora.
• MRMMIA: Membership Inference Attacks on Memory in Chat Agents — Chat agent memory is vulnerable to membership inference attacks, which MRMMIA effectively demonstrates across various access levels.
• [Checklist] Auditing AI for Deception — Behavioral AI alignment is a statistical illusion; internal mechanistic verification is crucial for safety.
• Data-Poisoning Audits for Causal Effect Estimation — Data-poisoning audits quantify causal estimate vulnerability to strategic record additions under capacity constraints.
• Medical records could be revealed by AI training-data vulnerability — Medical AI models are vulnerable to membership inference attacks, revealing sensitive patient data, with risk increasing with model capacity.
• Generalized Leverage Score for Scalable Assessment of Privacy Vulnerability — Individual data point privacy vulnerability correlates with its influence on a model, quantifiable via a generalized leverage score.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.