Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Software Development & Engineering · Depth: Expert, extended

Summary

An empirical study investigated whether Mixture-of-Experts (MoE) language models offer inference advantages on consumer and edge hardware, specifically an Apple M2 Pro and an NVIDIA Jetson Orin Nano 8 GB. Benchmarking OLMoE-1B-7B (1.3 B active, 6.9 B total parameters) against three dense baselines (Llama-3.2-1B, Qwen2.5-1.5B, Gemma-2-2B) using llama.cpp, researchers measured throughput, peak memory, and energy consumption. On the M2 Pro, OLMoE achieved 114.5 tokens/sec, about 10% slower than Llama-3.2-1B (127.3 tokens/sec) but faster than other dense models, with 4.5 GB peak memory. However, on the Jetson Orin Nano, OLMoE performed significantly worse, reaching 22.9 tokens/sec (31% slower than Llama-3.2-1B's 33.2 tokens/sec) and consuming 0.65 Joules/token, 2.1 times higher than Llama-3.2-1B's 0.31 Joules/token. Its 7.9 GB peak memory pushed the 8 GB Jetson to its limit. Analysis revealed routing overhead was under 9% of MoE-block compute on Jetson, indicating memory bandwidth, total parameter footprint, and expert dispatch are the primary bottlenecks, not routing arithmetic.

Key takeaway

For Machine Learning Engineers deploying LLMs on edge devices with limited memory, you should prioritize dense models over Mixture-of-Experts (MoE) architectures at the OLMoE-1B-7B scale. MoE models like OLMoE-1B-7B exhibit significantly lower throughput and higher energy consumption on 8 GB Jetson-class hardware, primarily due to their large total parameter footprint and memory bandwidth demands, not routing overhead. Consider further quantization or cache-aware routing for MoE if quality gains are critical.

Key insights

On bandwidth-bound edge hardware, MoE inference cost tracks total parameters, not active ones, negating expected efficiency gains.

Principles

• MoE's active-parameter advantage erodes on memory-constrained edge devices.
• Total parameter footprint dictates MoE memory requirements on edge.
• Routing arithmetic is not the primary bottleneck for MoE inference.

Method

The study empirically measured MoE and dense LLM inference on consumer and edge hardware, using llama.cpp, Q4_K_M quantization, and direct power rail readings for energy.

In practice

• Consider total parameters for MoE deployment on edge.
• Explore cache-aware routing strategies.
• Quantize MoE expert weights further for memory headroom.

Topics

• Mixture-of-Experts
• Edge AI Inference
• LLM Benchmarking
• Memory Bandwidth
• OLMoE-1B-7B
• llama.cpp

Code references

• Analytics-Everywhere-Lab/edge-moe
• ggerganov/llama.cpp

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• Understanding Mixture of Experts (MoE): How Modern LLMs Scale to Trillions of Parameters — Mixture of Experts enables LLMs to scale to trillions of parameters by activating only relevant sub-networks per token.
• assigning Moe to Gpus to reduce inference and memory usage — Optimizing MoE expert-to-GPU assignment using training logs is a critical challenge for LLM inference efficiency.
• Why MoE models get more from speculative decoding - Cohere — MoE sparsity and expert routing significantly enhance speculative decoding performance, especially at moderate batch sizes.
• A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference — Expert requests in MoE LLMs exhibit strong spatio-temporal correlations, enabling predictable prefetching for efficiency.
• The Hummingbird That Carries a Giant: Inside Colibrì, the 744B-Parameter Model Running on a Laptop… — Model size and memory requirements are not identical constraints for Mixture-of-Experts models.
• Mixture of Experts (MoEs) in Transformers — MoEs offer high model capacity with reduced inference costs by activating only a subset of experts per token.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.