HASA: Subnet Allocation for Compute-Constrained Model-Heterogeneous Federated Learning

2026-05-30 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Expert, quick

Summary

HASA (Heterogeneity-Aware Subnet Allocation) is a novel train-only rule designed for compute-constrained, model-heterogeneous federated learning environments. It addresses the challenge of varying client resources and data distributions by assigning subnet widths based on client heterogeneity scores derived from local training data, while adhering to a fixed size-weighted compute budget. On an article-title next-word prediction benchmark with seven clients, HASA significantly improved unweighted mean client test accuracy from 13.82 percent to 14.32 percent over uniform allocation across 10 matched seeds, and enhanced worst-client accuracy on average. In direct comparisons, HASA achieved the strongest worst-client and tail-client accuracy against representative partial-training baselines. An ablation study revealed that allocating smaller subnets to more heterogeneous clients negatively impacts both mean and tail performance, and a cross-domain image-classification study indicated that HASA's effectiveness relies on the heterogeneity score accurately reflecting clients' need for additional model width.

Key takeaway

For MLOps Engineers deploying federated learning models in resource-constrained, heterogeneous environments, you should consider implementing heterogeneity-aware subnet allocation strategies like HASA. This approach can significantly improve overall model performance and, critically, enhance worst-client and tail-client accuracy, which is vital for equitable user experience. Ensure your heterogeneity scoring accurately reflects a client's actual need for model complexity to maximize the benefits of dynamic subnet sizing.

Key insights

HASA improves federated learning performance by dynamically allocating subnet widths based on client data heterogeneity.

Principles

• Subnet allocation should account for statistical heterogeneity.
• Smaller subnets for heterogeneous clients degrade performance.
• Heterogeneity scores must reflect model width needs.

Method

HASA computes client heterogeneity scores from local training data and assigns subnet widths accordingly, while maintaining a fixed size-weighted compute budget.

In practice

• Implement heterogeneity scoring for client data.
• Dynamically adjust subnet sizes based on client needs.
• Benchmark allocation policies against worst-client accuracy.

Topics

• Federated Learning
• Subnet Allocation
• Model Heterogeneity
• Compute Constraints
• Edge AI
• Machine Learning Performance

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Split Federated Learning Architectures for High-Accuracy and Low-Delay Model Training — Optimizing model partitioning in Split Federated Learning improves accuracy, reduces delay, and lowers communication overhead.
• CRAFT: Conflict-Resolved Aggregation for Federated Training — CRAFT resolves federated learning update conflicts via geometric projection, improving global accuracy and client fairness.
• EvoCSFL: Surrogate-Assisted Evolutionary Client Selection for Efficient and Robust Federated Learning — EvoCSFL uses a surrogate-assisted evolutionary algorithm to optimize client selection in federated learning for improved efficiency and robustness.
• Optimal Complexity in Byzantine-Robust Distributed Stochastic Optimization with Data Heterogeneity — Tight lower bounds for Byzantine-robust distributed stochastic optimization are established and matched by new methods.
• Federated Nested Learning: Collaborative Training of Self-Referential Memories for Test-Time Adaptation — FedNL enables federated models to learn adaptive memory update rules, not just static weights, for robust test-time adaptation.
• Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Models — HyperLoRA amortizes federated LoRA adaptation via hypernetwork-driven initialization and product-space aggregation, resolving bias and inefficiency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.