Towards Privacy-Preserving Federated Prompt Tuning under Data Heterogeneity: A Subspace-Decomposed Expert Approach

2026-07-23 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

FedSEPT introduces a novel privacy-preserving Federated Subspace-decomposed Expert Prompt Tuning approach, designed to enhance collaborative adaptation of vision-language models (VLMs). Existing Federated Prompt Tuning (FPT) methods, which use a split-prompt design with local differential privacy, often struggle with a single shared prompt over-smoothing diverse knowledge, impacting personalization and generalization. While multi-expert prompts (MEPs) offer diversity, they increase communication costs and differential privacy noise. FedSEPT addresses this through Subspace-decomposed Expert Modeling (SEM), parameterizing multiple prompt experts with shared low-rank factors, a fixed public basis, and private residuals. This confines communication and DP perturbation to a compact factor space, enabling direct server aggregation. It also features Instance-aware Expert Fusion (IEF) for adaptive, on-device expert combination and efficient logit-level fusion. Experiments on 11 heterogeneous benchmarks show FedSEPT achieves a better balance between local adaptation and global generalization under the same privacy constraints.

Key takeaway

For Machine Learning Engineers developing federated learning solutions for vision-language models, FedSEPT provides a robust approach to balance personalization and generalization while maintaining privacy. If you are struggling with communication costs or differential privacy noise from multi-expert prompts, consider implementing subspace-decomposed expert modeling. This method allows for more effective adaptation across diverse datasets without over-smoothing knowledge, offering a superior trade-off compared to existing baselines.

Key insights

FedSEPT uses subspace-decomposed experts and instance-aware fusion for privacy-preserving federated prompt tuning, balancing personalization and generalization.

Principles

• Shared low-rank factors reduce communication.
• Fixed public basis enables direct server aggregation.
• Adaptive expert fusion improves personalization.

Method

FedSEPT employs Subspace-decomposed Expert Modeling (SEM) with shared low-rank factors and private residuals for prompt parameterization. Instance-aware Expert Fusion (IEF) adaptively combines experts via on-device routing and logit-level fusion using cached text features.

In practice

• Adapt VLMs collaboratively with privacy.
• Improve personalization on diverse datasets.
• Reduce communication overhead in FPT.

Topics

• Federated Learning
• Prompt Tuning
• Vision-Language Models
• Differential Privacy
• Data Heterogeneity
• Expert Models

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Experts — FC-MoE resolves expert conflicts in federated MoE LLM fine-tuning, improving stability and performance in non-IID settings.
• FedOPAL: One-Shot Federated Learning via Analytic Visual Prompt Tuning — FedOPAL uses visual prompts to rectify non-IID data for efficient one-shot analytical federated learning with zero server-side cost.
• Collaborative Synthetic Data Generation for Knowledge Transfer in Federated Learning — FedKT-CSD enables private, one-shot federated learning by collaboratively generating synthetic data from client latent statistics.
• Enabling privacy-preserving AI training on everyday devices — FTTE accelerates federated learning on resource-constrained edge devices by optimizing memory, communication, and update synchronization.
• Decentralized Federated Learning by Partial Message Exchange — PaME improves DFL communication, privacy, and accuracy by exchanging sparse, randomly selected model coordinates.
• FedCVESA: Taking Away Training Data in Federated Learning via Correlation Value Encoding and Segmented Aggregation — Federated learning can be exploited by malicious servers to embed and extract private training data.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.