Class-Grouped Normalized Momentum and Faster Hyperparameter Exploration to Tackle Class Imbalance in Federated Learning

2026-07-01 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

FedCGNM (Federated Class-Grouped Normalized Momentum) is a client-side optimizer designed to address class imbalance in federated learning (FL), a critical challenge where underrepresented classes perform poorly due to privacy and heterogeneity constraints. This method partitions classes into groups based on minimum within-group variance, maintains a momentum for each group, normalizes each group's momentum to unit length, and sums these normalized momentums for the update direction. This approach effectively equalizes gradient magnitude across majority and minority groups while mitigating noise in rare-class gradients. The work also provides a theoretical convergence analysis. Additionally, FedHOO, an X-armed-bandit (XAB) based algorithm, is introduced for efficient hyperparameter optimization of resampling rates in small-client FL regimes, exploiting federated parallelism. Empirical evaluations on four public long-tailed benchmarks and a proprietary chip-defect dataset confirm FedCGNM's consistent outperformance, with FedHOO providing further gains in small-scale federations.

Key takeaway

For machine learning engineers addressing class imbalance in federated learning, you should consider integrating FedCGNM into your client-side optimizers. This can significantly improve predictive performance for underrepresented classes on long-tailed datasets. Furthermore, if you are operating in small-client federations, utilize FedHOO to efficiently explore and optimize resampling rates, ensuring robust model convergence and enhanced overall system performance.

Key insights

FedCGNM tackles federated learning class imbalance via client-side class-grouped normalized momentum, enhanced by FedHOO for hyperparameter optimization.

Principles

• Class imbalance in FL requires privacy-preserving client-side solutions.
• Normalizing group gradients mitigates noise and magnitude disparities.
• X-armed bandit algorithms can optimize FL hyperparameters efficiently.

Method

FedCGNM partitions classes by variance, maintains normalized momentum per group, and sums these for updates. FedHOO uses X-armed-bandit to evaluate resampling rates with federated parallelism.

In practice

• Implement FedCGNM as a client-side FL optimizer.
• Utilize FedHOO for efficient hyperparameter tuning in small federations.
• Consider class grouping for imbalanced data in distributed settings.

Topics

• Federated Learning
• Class Imbalance
• Gradient Momentum
• Hyperparameter Optimization
• X-armed Bandit
• Client-side Optimization

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• FedIDM: Achieving Fast and Stable Convergence in Byzantine Federated Learning through Iterative Distribution Matching — FedIDM uses distribution matching and condensed data to achieve fast, stable, and robust federated learning convergence.
• FOSC-X: An Extended Framework for Optimal Local Cuts and Non-Horizontal Cluster Selection from Clustering Hierarchies — FOSC-X extends hierarchical clustering to extract top-M optimal flat solutions with flexible cluster-count constraints via dynamic programming.
• When Generator Replay Degrades: Projected Rehearsal Orchestration for Heterogeneous Federated Class-Incremental Learning — Projected rehearsal orchestration (PRO) and PRO-MAX improve federated class-incremental learning by replacing input-space synthesis with server-light projected memories.
• Split Federated Learning Architectures for High-Accuracy and Low-Delay Model Training — Optimizing model partitioning in Split Federated Learning improves accuracy, reduces delay, and lowers communication overhead.
• Customer Churn Prediction on Structured Data Using FT-Transformer and Stacking Ensembles — Hybrid models combining transformers and tree ensembles offer superior, calibrated churn prediction on tabular data.
• Federated orthogonal learning for detection of liver lesions from multi-phase contrast-enhanced CT images — FedOG enhances federated learning for medical image segmentation by using orthogonal gradient decomposition to counter data heterogeneity and incomplete phases.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.