Predictability as a Fine-Grained Measure for Privacy

2026-06-18 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Expert, quick

Summary

Privacy via predictability is a novel framework introduced to measure privacy leakage, offering a fine-grained alternative or complement to Differential Privacy (DP). Unlike DP's worst-case guarantees, this framework explicitly incorporates an attacker's core knowledge, a compromised dataset portion from a stochastic process, and a specified query family. Predictability quantifies privacy leakage as the incremental gain in an attacker's ability to predict sensitive information about unknown individuals, beyond what's inferable from compromised data. The framework demonstrates that predictability and DP are generally incomparable, though in a worst-case scenario where all but one individual is compromised and all binary queries are sensitive, predictability implies mutual-information DP. A general framework using the generalized method of moments (GMM) is presented for analyzing asymptotic predictability under stationary, ergodic, mixing processes, leading to a predictability-calibrated output perturbation scheme for ERM. This approach can be used alongside DP for enhanced privacy control.

Key takeaway

For AI Security Engineers designing privacy-preserving systems, you should consider integrating "privacy via predictability" alongside Differential Privacy. This framework tailors privacy guarantees by accounting for specific attacker knowledge and query types, potentially reducing the privacy-accuracy tradeoff. Use its predictability-calibrated output perturbation scheme for ERM to achieve finer-grained control over sensitive information leakage in your models.

Key insights

Privacy via predictability offers a fine-grained privacy metric by quantifying an attacker's incremental prediction gain based on specific knowledge and queries.

Principles

• Predictability and DP are generally incomparable.
• Predictability implies mutual-information DP in worst-case.
• Incorporate attacker knowledge for fine-grained privacy.

Method

A general framework using the generalized method of moments (GMM) analyzes asymptotic predictability for data generated by stationary, ergodic, mixing processes, deriving a predictability-calibrated output perturbation scheme for ERM.

In practice

• Use alongside DP for fine-grained control.
• Tailor privacy metrics to specific sensitive data.
• Calibrate output perturbation for ERM.

Topics

• Predictability
• Differential Privacy
• Privacy Metrics
• Attacker Models
• Generalized Method of Moments
• Empirical Risk Minimization

Best for: Research Scientist, AI Scientist, AI Security Engineer

Related on AIssential

• Benchmarking Empirical Privacy Protection for Adaptations of Large Language Models — Distribution shifts critically impact practical privacy in differentially private LLM adaptations, often undermining theoretical guarantees.
• How probability models protect privacy — Differential privacy uses calibrated randomness to protect individual data while enabling accurate aggregate analysis.
• Gaussian DP for Reporting Differential Privacy Guarantees in Machine Learning — Gaussian Differential Privacy (GDP) offers a single, comparable parameter (μ) for reporting ML privacy guarantees, improving upon (ε,δ)-DP.
• Generalized Leverage Score for Scalable Assessment of Privacy Vulnerability — Individual data point privacy vulnerability correlates with its influence on a model, quantifiable via a generalized leverage score.
• Differential Privacy in 30 seconds — Differential Privacy protects individual data by clipping gradients and adding noise during model training.
• Differentially Private Conformal Prediction — DPCP offers statistically efficient, end-to-end differentially private conformal prediction without data splitting.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.