Decentralized Federated Learning by Partial Message Exchange

2026-07-09 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy, Cloud Computing & IT Infrastructure · Depth: Expert, extended

Summary

PaME (DFL by Partial Message Exchange) is a novel decentralized federated learning (DFL) algorithm designed to overcome fundamental challenges in large-scale heterogeneous networks, including data heterogeneity, restrictive theoretical assumptions, and degraded convergence. The algorithm achieves substantial reductions in communication costs and preserves a high level of privacy without sacrificing accuracy by allowing only randomly selected sparse coordinates to be exchanged between neighbor nodes. PaME is rigorously proven to converge at a linear rate under mild assumptions, such as locally Lipschitz continuous gradients and a doubly stochastic communication matrix, effectively handling heterogeneous data and dynamic communication graphs. Comprehensive numerical experiments demonstrate PaME's superior performance in communication efficiency, convergence speed, and robustness to data heterogeneity compared to several representative DFL algorithms on various tasks, including linear/logistic regression, Fashion-MNIST (CNN), and CIFAR-10 (ResNet-20).

Key takeaway

For ML Engineers deploying DFL in resource-constrained or privacy-sensitive environments, PaME offers a robust solution. You should consider implementing its partial message exchange mechanism to significantly reduce communication overhead (by up to 80%) and enhance privacy, while maintaining or improving model accuracy and convergence speed, even with heterogeneous data and dynamic network topologies. This approach provides a more favorable trade-off than existing DFL methods.

Key insights

PaME improves DFL communication, privacy, and accuracy by exchanging sparse, randomly selected model coordinates.

Principles

• Sparse coordinate exchange reduces communication overhead.
• Randomness in message exchange enhances privacy preservation.
• Unbiased estimation is crucial for accurate sparse parameter aggregation.

Method

PaME alternates local model updates with a Partial Message Exchange (PME) mechanism, where nodes transmit sparse, randomly selected coordinates, which are then averaged using an unbiased estimator.

In practice

• Implement partial message exchange to reduce DFL communication by up to 80%.
• Utilize heterogeneous communication periods for more realistic training deployments.
• Adjust transmission and participation rates to balance convergence speed and communication load.

Topics

• Decentralized Federated Learning
• Partial Message Exchange
• Communication Efficiency
• Privacy Preservation
• Model Convergence
• Data Heterogeneity

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Statistical Limits and Efficient Algorithms for Differentially Private Federated Learning — New methods improve federated learning accuracy and communication efficiency while preserving differential privacy.
• Information-Theoretically Secure Aggregation for Lightweight Federated Learning: Resilient to Dropouts and Adversaries — Secure aggregation for sign-based FL can be lightweight and information-theoretically robust against dropouts and adversaries.
• C2FL: Clustered Continual Federated Learning under Spatial and Temporal Drift — C2FL enables robust federated learning in mobile, privacy-sensitive systems by combining spatial clustering with temporal drift adaptation.
• Towards Privacy-Preserving Federated Prompt Tuning under Data Heterogeneity: A Subspace-Decomposed Expert Approach — FedSEPT uses subspace-decomposed experts and instance-aware fusion for privacy-preserving federated prompt tuning, balancing personalization and generalization.
• Federated Learning over Blockchain-Enabled Cloud Infrastructure — Integrating Federated Learning with blockchain in cloud-edge environments enhances data privacy and security.
• Beyond Weights and Gradients: A Taxonomy of Federated Learning Messages — Federated learning messages extend beyond weights, requiring a taxonomy for utility and privacy trade-offs.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.