Entropy-Guided Tensor Compression for Multimodal Federated Learning on Edge Devices

2026-07-09 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Internet of Things (IoT) & Connected Devices, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

MESH-FL, an entropy-guided matrix product state (MPS) update-compression framework, addresses challenges in multimodal federated learning (FL) on resource-constrained edge devices. It tackles issues where clients have varying sensing capabilities and computational capacities, and existing compression schemes apply uniform policies. MESH-FL estimates the spectral entropy of each layer-wise update via truncated singular value decomposition (SVD) and adaptively allocates MPS compression ranks across layers, modalities, and devices, adhering to per-client payload budgets. Experiments on a 15-node heterogeneous Raspberry Pi 4/5 cluster with the AV-MNIST dataset demonstrated MESH-FL achieved up to 56.8x compression, surpassed the uncompressed FedAvg baseline in final accuracy by up to 2.01% (reaching 97.73% vs 95.72%), and reduced total transmitted data to convergence by up to 66x (from 33.63 GB to 0.51 GB for heavy compression). The framework also showed faster convergence to accuracy thresholds.

Key takeaway

For AI Architects designing federated learning systems for multimodal data on heterogeneous edge devices, you should consider implementing entropy-guided tensor compression. MESH-FL demonstrates that adaptively allocating compression ranks based on spectral entropy can significantly reduce communication costs by up to 66x and even improve model accuracy by 2.01% compared to uncompressed baselines, accelerating convergence on resource-constrained hardware. This approach optimizes resource utilization and enhances learning performance.

Key insights

Spectral entropy guides adaptive compression rank allocation in multimodal federated learning, improving efficiency and accuracy on heterogeneous edge devices.

Principles

• Higher spectral entropy necessitates higher reconstruction rank.
• Adaptive compression improves accuracy in non-IID FL.
• Truncated SVD can efficiently estimate spectral entropy.

Method

MESH-FL estimates layer-wise spectral entropy via truncated SVD, then allocates MPS ranks adaptively across layers, modalities, and devices under per-client payload budgets, followed by MPS compression and layer-wise aggregation.

In practice

• Implement entropy-guided rank allocation for FL model updates.
• Use MPS compression for efficient tensor decomposition.
• Deploy on heterogeneous edge clusters like Raspberry Pi 4/5.

Topics

• Federated Learning
• Tensor Compression
• Multimodal AI
• Edge Devices
• Matrix Product State
• Spectral Entropy

Code references

• Jakobovski/free-spoken-digit-dataset

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Entropy-Guided Tensor Compression for Multimodal Federated Learning on Edge Devices — Adaptive, entropy-guided tensor compression significantly boosts federated learning efficiency and accuracy on edge devices.
• CoCo-Fed: A Unified Framework for Memory- and Communication-Efficient Federated Learning at the Wireless Edge — CoCo-Fed unifies local memory efficiency and global communication reduction in federated learning for wireless edge networks.
• SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices — SigmaScale enhances SVD-based LLM compression by learning activation-aware scaling matrices, improving efficiency.
• A Typed Tensor Language for Federated Learning — The language formalizes federated learning computations, enabling fixed-dimensional shared state communication for efficiency.
• Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers — SpecTraL uses spectral transformation and random matrix theory to optimize federated LoRA for ViTs, improving efficiency and accuracy without manual rank tuning.
• M$^2$FedAQI: Multimodal Federated Learning for Air Quality Prediction on Heterogeneous Edge Devices — M^2FedAQI offers a multimodal federated learning solution for accurate, private, and efficient air quality prediction on edge devices.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.