MEGO: Learning Mixture-of-Experts for General-Purpose Binary Optimization

2024-05-29 · Source: cs.NE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Optimization & Evolutionary Computing · Depth: Expert, short

Summary

MEGO is a novel general-purpose neural optimizer designed for binary optimization problems in black-box settings, aiming for broad applicability with minimal customization. This system employs a mixture-of-experts architecture, trained without specific domain knowledge, and utilizes a dynamic routing policy to activate the most relevant expert models for generating high-quality solutions to new problem instances. MEGO demonstrates strong generalization capabilities across six diverse problem classes, encompassing both classic and real-world applications. Notably, when trained exclusively on classic problems, MEGO effectively generalizes to unseen and complex real-world scenarios, significantly surpassing widely-used general-purpose optimizers in both solution quality and efficiency. Furthermore, MEGO introduces a computational method for quantifying similarity and classifying optimization problems, offering an alternative to traditional analysis-based approaches. The paper, comprising 33 pages and 7 figures, was last revised on July 8, 2026 (v2).

Key takeaway

For Machine Learning Engineers or Research Scientists developing solutions for complex discrete optimization, MEGO offers a compelling alternative to traditional general-purpose optimizers. You should consider integrating MEGO for its demonstrated ability to generalize across diverse binary optimization problems, even unseen real-world scenarios, delivering superior solution quality and efficiency. Evaluate its computational approach for problem classification to gain new insights into problem relationships.

Key insights

MEGO uses a mixture-of-experts and dynamic routing to generalize across diverse binary optimization problems, outperforming traditional optimizers.

Principles

• Mixture-of-experts can achieve broad generalization without domain knowledge.
• Dynamic routing policies enhance adaptability to new problem instances.
• Computational similarity metrics offer new problem classification methods.

Method

MEGO trains a mixture-of-experts without domain knowledge. It then uses a routing policy to dynamically select and activate experts to generate solutions for new binary optimization problems.

In practice

• Apply MEGO to diverse black-box binary optimization tasks.
• Use MEGO for classifying optimization problems by similarity.
• Evaluate MEGO against existing optimizers for efficiency gains.

Topics

• Mixture-of-Experts
• Binary Optimization
• Neural Optimizers
• Generalization
• Problem Classification

Best for: AI Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts — ProbMoE offers a differentiable probabilistic routing solution for MoE models, improving expert utilization and performance.
• MMAO-Cls: Metabolic Multi-Agent Optimization for Joint Feature Selection and Classifier Tuning — MMAO-Cls integrates multi-agent optimization for joint feature selection and hyperparameter tuning, yielding compact models.
• One-Step Generalization Ratio Guided Optimization for Domain Generalization — GENIE optimizer uses One-Step Generalization Ratio (OSGR) to balance parameter contributions for robust domain generalization.
• An Evolutionary Algorithm with Probabilistic Annealing for Large-scale Sparse Multi-objective Optimization — PAMEA uses dual-entropy probability vectors and annealing-based clustering to balance exploration and exploitation in large-scale sparse multi-objective optimization.
• Multi-Rate Mixture of Experts for Accelerating Liquid Neural Network Training — Multi-Rate Mixture-of-Experts enhances Liquid Neural Networks for complex, multi-scale time-series modeling.
• Subject to: Bissan Ghaddar — Resilience and adaptability are crucial for navigating complex academic and professional paths, especially in optimization.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.NE updates on arXiv.org.