Entropy-Guided Tensor Compression for Multimodal Federated Learning on Edge Devices

2026-07-07 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Internet of Things (IoT) & Connected Devices · Depth: Expert, quick

Summary

MESH-FL is an entropy-guided matrix product state (MPS) update-compression framework designed for modality-heterogeneous federated learning (FL) on resource-constrained edge devices. It addresses the limitations of existing uniform compression schemes by adaptively allocating MPS compression ranks across layers, modalities, and devices. MESH-FL estimates the spectral entropy of each layer-wise update via truncated singular value decomposition. It then allocates higher MPS compression ranks to layers exhibiting higher spectral entropy. This approach solves a convex surrogate rank-allocation problem, preserves monotonicity, and achieves convergence with a compression-dependent error term. Experiments on a 15-node heterogeneous Raspberry Pi 4/5 cluster demonstrated MESH-FL achieving up to 56.8× compression. It surpassed the uncompressed FedAvg baseline in final accuracy by up to 2.01%. Total transmitted data to reach convergence was reduced by up to 66×.

Key takeaway

For Machine Learning Engineers deploying multimodal federated learning on resource-constrained edge devices, you should consider implementing adaptive tensor compression. MESH-FL's entropy-guided rank allocation method can drastically reduce data transmission by up to 66×. It can also improve model accuracy by over 2.01% compared to uncompressed baselines. This approach allows your models to converge faster and more efficiently on heterogeneous hardware like Raspberry Pi clusters. It optimizes your deployment costs and performance.

Key insights

Adaptive, entropy-guided tensor compression significantly boosts federated learning efficiency and accuracy on edge devices.

Principles

• Higher spectral entropy needs higher reconstruction rank.
• Adaptive rank allocation improves FL convergence.

Method

Estimate layer-wise spectral entropy via truncated SVD, then adaptively allocate MPS compression ranks based on entropy under per-client payload budgets.

In practice

• Implement adaptive compression for multimodal FL.
• Use spectral entropy to guide rank allocation.
• Deploy on Raspberry Pi-like edge clusters.

Topics

• Federated Learning
• Tensor Compression
• Edge AI
• Multimodal Models
• Matrix Product State
• Spectral Entropy
• Raspberry Pi

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• A Typed Tensor Language for Federated Learning — The language formalizes federated learning computations, enabling fixed-dimensional shared state communication for efficiency.
• SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices — SigmaScale enhances SVD-based LLM compression by learning activation-aware scaling matrices, improving efficiency.
• Exploiting Correlations in Federated Learning: Opportunities and Practical Limitations — Federated learning compression schemes exploit structural, temporal, or spatial correlations, which vary significantly by task and model.
• M$^2$FedAQI: Multimodal Federated Learning for Air Quality Prediction on Heterogeneous Edge Devices — M^2FedAQI offers a multimodal federated learning solution for accurate, private, and efficient air quality prediction on edge devices.
• Federated Learning of Binary Neural Networks: Enabling Low-Cost Inference — Federated Learning (FL) preserves privacy by distributing training across devices, but deploying computationally intensive Deep Neural Networks (DNNs) for inference on low-powered edge devices present…
• Efficient frequent directions algorithms for approximate decomposition of matrices and higher-order tensors — New frequent directions algorithms enhance low-rank matrix and tensor approximations for efficiency and accuracy.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.