Benchmarking Optimizers for MLPs in Tabular Deep Learning

2026-04-16 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, quick

Summary

A new study systematically benchmarks various optimizers for training Multi-Layer Perceptrons (MLPs) on tabular datasets, a common practice in supervised deep learning. The research evaluates Noptimizers optimizers across Ndatasets tabular datasets using a consistent experimental protocol. The primary finding indicates that the Muon optimizer consistently surpasses AdamW in performance, positioning it as a robust alternative for practitioners and researchers, provided its training efficiency overhead is manageable. Additionally, the study identifies exponential moving average (EMA) of model weights as a straightforward yet effective technique to enhance AdamW's performance on vanilla MLPs, though its impact varies across different model configurations.

Key takeaway

For AI Engineers and Research Scientists developing or deploying MLP-based models on tabular data, you should evaluate the Muon optimizer as a primary alternative to AdamW. While Muon offers superior performance, assess its training efficiency overhead against your project's resource constraints. Additionally, consider integrating exponential moving average (EMA) with AdamW for a simple performance boost, particularly with vanilla MLP architectures.

Key insights

Muon optimizer consistently outperforms AdamW for tabular MLP training, offering a strong practical choice.

Principles

• Optimizer choice significantly impacts tabular deep learning performance.
• EMA can improve AdamW on vanilla MLPs.

Method

Benchmarking Noptimizers optimizers on Ndatasets tabular datasets for MLP-based models under a shared supervised learning protocol to evaluate performance and efficiency.

In practice

• Consider Muon as a primary optimizer for tabular MLPs.
• Implement EMA with AdamW for potential performance gains.

Topics

• Tabular Deep Learning
• Multi-Layer Perceptrons
• Optimizers Benchmarking
• Muon Optimizer
• AdamW

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Data Scientist

Related on AIssential

• The First Optimizer to Challenge Adam in a Decade Just Cut Training Costs in Half — Muon optimizer offers significant compute efficiency gains, challenging Adam's long-standing dominance in deep learning.
• [Webinar Recording] Zero Tuning. State-of-the-Art Predictions. How TabH2O Changes Tabular ML — Tab H2O is a pre-trained, transformer-based tabular foundation model offering high-accuracy predictions without tuning.
• Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate — Maximizing the embedding layer learning rate is crucial for effective hyperparameter transfer in large language model training.
• XGBoost Explained: Why Gradient Boosting Became a Machine Learning Powerhouse — XGBoost optimized gradient boosting for scale and efficiency, making it a powerhouse for tabular data.
• Muon Learns More Robust and Transferable Features than Adam — Muon optimizer yields more robust and transferable features than Adam and SGD across diverse model architectures and tasks.
• This Simple Optimizer Is Revolutionizing How We Train AI [Muon] — Muon Optimizer enhances training efficiency and stability by orthogonalizing momentum and clipping attention logits.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.