CoSMoEs: Compact Sparse Mixture of Experts

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, medium

Summary

CoSMoEs, or Compact Sparse Mixture of Experts, is a novel architecture introduced for on-device inference, specifically addressing the challenges of quality, memory, and latency in smaller-scale applications of Sparse Mixture of Expert (MoE) models. While MoE models are widely used at large scales, their application on devices has been under-explored. The research, presented at the 4th Workshop on Advances in Language and Vision Research (ALVR) in July 2026, demonstrates that MoE architectures can outperform dense models at on-device scale through fair evaluation. CoSMoEs further enhances performance by proposing weight-decomposed experts. To tackle the large parameter count, it improves expert offloading efficiency using a novel training-time loss, thereby reducing inference latency for on-device deployment. This work, detailed on pages 46–56, aims to make powerful MoE models practical for resource-constrained environments.

Key takeaway

For Machine Learning Engineers optimizing models for on-device deployment, CoSMoEs presents a viable path to utilize Sparse Mixture of Experts. You should consider integrating weight-decomposed experts and the proposed training-time loss to improve model quality and reduce inference latency. This approach allows you to deploy more powerful MoE architectures on resource-constrained devices, overcoming traditional memory and latency bottlenecks.

Key insights

CoSMoEs enables efficient on-device MoE inference by improving quality, memory, and latency through architectural and training innovations.

Principles

• MoE architectures can surpass dense models on-device.
• Weight decomposition enhances MoE performance.
• Training-time loss improves expert offloading.

Method

CoSMoEs employs weight-decomposed experts and a novel training-time loss to enhance expert offloading efficiency, reducing parameter count and inference latency for on-device MoE deployment.

In practice

• Deploy MoE models on resource-constrained devices.
• Implement weight-decomposed expert layers.
• Utilize training-time loss for offloading.

Topics

• Sparse Mixture of Experts
• On-device Inference
• Model Compression
• Weight Decomposition
• Inference Latency
• Memory Efficiency

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models — Fine-tuning MoE models to prefer fewer experts significantly reduces memory transfer overhead during inference.
• Extreme Co-Design for Efficient Tokenomics and AI at Scale — Extreme co-design across the entire AI stack is crucial for efficient, scalable real-time reasoning and MoE models.
• NEW Mixture of Expert MoE: Spectral Decomposition in Orthogonal Subspaces — AI reasoning models exhibit significant vulnerability to multi-turn attacks due to spectral bias in MoE gating.
• GPT-OSS 20B: A Sparse MoE Pretraining Benchmark for MLPerf Training v6.0 — GPT-OSS 20B offers a stable, accessible benchmark for sparse Mixture-of-Experts LLM pretraining in MLPerf Training v6.0.
• Accelerating Mixture-of-Experts Execution with FarSkip-Collective Models — FarSkip-Collective accelerates MoE models by overlapping computation and communication, eliminating idle GPU time during distributed execution.
• Does this idea sound fun? [R] — A PoC demonstrates inference-time learning in MoE by using specialized experts to update sibling expert weights.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.