Differentially Private Model Merging

2026-04-24 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Expert, extended

Summary

This work introduces two post-processing techniques, random selection (RS) and linear combination (LC), for differentially private model merging. The goal is to generate models that satisfy varying target differential privacy (DP) requirements without additional training, given a set of existing models trained on the same dataset with different privacy/utility tradeoffs. The authors provide privacy accounting for both approaches using Rényi DP and privacy loss distributions. A case study on private mean estimation theoretically establishes the superiority of linear combination over random selection. Empirical validation on synthetic and real-world datasets, including MNIST and CIFAR-10, demonstrates that both RS and LC improve privacy/utility tradeoffs compared to naive privacy accounting. The linear combination method requires structural assumptions on the training algorithms, such as DP-SGD, to achieve meaningful privacy gains beyond joint release.

Key takeaway

For research scientists developing machine learning applications with evolving privacy needs, consider implementing differentially private model merging techniques like Random Selection or Linear Combination. These methods allow you to adapt existing private models to new privacy constraints efficiently, avoiding costly retraining. Prioritize Linear Combination if your base models were trained with DP-SGD, as it offers tighter privacy bounds and better utility. This approach can significantly streamline model deployment and compliance with dynamic regulations.

Key insights

Merge existing differentially private models to meet dynamic privacy requirements without retraining.

Principles

• Privacy requirements can change dynamically.
• Retraining for new DP levels is computationally expensive.
• Model merging can adapt privacy post-training.

Method

Two data-independent post-processing techniques: random selection (RS) and linear combination (LC). RS samples a model based on a probability distribution; LC linearly averages models. Both optimize mixing coefficients for utility under privacy constraints.

In practice

• Use RS for models with known RDP/PLD profiles.
• Apply LC when input models are DP-SGD trained.
• Evaluate mixtures on public data for optimal utility.

Topics

• Differential Privacy
• Model Merging
• Rényi Differential Privacy
• Privacy Loss Distributions
• Random Selection

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• DP-CDA: An Algorithm for Enhanced Privacy Preservation in Dataset Synthesis Through Randomized Mixing — DP-CDA generates privacy-preserving synthetic datasets by class-specific random mixing and Gaussian noise, optimizing utility and privacy.
• Private Learning with Public Feature Conditioning — Cond-DP uses public features to condition DPSGD, accelerating private regression convergence without extra privacy cost.
• Benchmarking Empirical Privacy Protection for Adaptations of Large Language Models — Distribution shifts critically impact practical privacy in differentially private LLM adaptations, often undermining theoretical guarantees.
• Sequential Auditing for f-Differential Privacy — Sequential f-DP auditors adaptively determine sample sizes, significantly reducing auditing costs and improving privacy violation detection.
• Differential Privacy in 30 seconds — Differential Privacy protects individual data by clipping gradients and adding noise during model training.
• Secure and Privacy-Preserving Vertical Federated Learning — A novel framework enhances vertical federated learning privacy via distributed aggregation and secure multiparty computation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.