Kernel Selection is Model Selection: A Unified Complexity-Penalized Approach for MMD Two-Sample Tests

2026-05-11 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

The paper "Kernel Selection is Model Selection: A Unified Complexity-Penalized Approach for MMD Two-Sample Tests" introduces Complexity-Penalized MMD (CP-MMD), a novel criterion for nonparametric two-sample testing that addresses the critical challenge of data-driven kernel selection. Existing Maximum Mean Discrepancy (MMD) methods either overfit due to violating i.i.d. assumptions or cannot scale to continuous kernel search spaces like deep kernels. CP-MMD reframes kernel selection as a model selection problem, applying a two-sample uniform concentration inequality to derive a penalty term that accounts for optimization complexity. This enables grid-free maximization over continuous parametric classes, including scalar bandwidths, polynomial features, and deep network parameters. The method is proven to maximize true test power while ensuring unconditional Type-I validity, outperforming state-of-the-art methods like the median heuristic, MMDAgg, and Liu's ratio criterion across linear, polynomial-feature, and deep kernel regimes, and demonstrating $B$-free deployment costs.

Key takeaway

For AI Engineers and Research Scientists developing or deploying two-sample tests, CP-MMD offers a robust and efficient solution for data-driven kernel selection. Its ability to perform grid-free optimization over continuous kernel spaces, including deep neural networks, resolves critical overfitting and scalability issues inherent in prior methods. You should consider integrating CP-MMD to maximize test power and ensure Type-I validity, especially when working with complex, high-dimensional data or deep kernel architectures, as it significantly reduces deployment costs compared to aggregation methods.

Key insights

CP-MMD unifies kernel selection as a complexity-penalized model selection problem for robust two-sample testing.

Principles

• Kernel selection is a model selection problem.
• Data-driven kernel optimization requires complexity penalization.
• Uniform concentration inequalities can bound post-optimization MMD.

Method

CP-MMD maximizes empirical MMD minus a calibrated complexity penalty $\widehat{C}_1\cdot\widetilde{G}(h)$, where $\widetilde{G}(h)$ is a spectral-norm bound on kernel search space complexity, enabling grid-free optimization.

In practice

• Use CP-MMD for continuous kernel search spaces.
• Calibrate $\widehat{C}_1$ using null permutations for optimal performance.
• CP-MMD offers $B$-free deployment cost for deep kernels.

Topics

• Maximum Mean Discrepancy
• Kernel Selection
• Two-Sample Testing
• Complexity-Penalized MMD
• Uniform Concentration Inequality

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Data Scientist

Related on AIssential

• Beyond Cross-Validation: Adaptive Parameter Selection for Kernel-Based Gradient Descents — This paper proposes a novel Hybrid Selection Strategy (HSS) for Kernel-Based Gradient Descent (KGD) algorithms, which integrates bias-variance analysis with the splitting method for adaptive parameter selection.
• Computational-Statistical Trade-off in Kernel Two-Sample Testing with Random Fourier Features — Approximated MMD tests with Random Fourier Features can match full MMD power at sub-quadratic time with careful feature selection.
• To select or not to select? — Predictively consistent priors enable complex models to outperform or match simpler selected models without explicit model selection.
• On the Robustness of Kernel Goodness-of-Fit Tests — Existing kernel goodness-of-fit tests lack robustness, necessitating a new framework for practical model validation.
• Learning What Matters: Probabilistic Task Selection via Mutual Information for Model Finetuning — TaskPGM optimizes LLM fine-tuning data mixtures by balancing task representativeness and diversity through an energy-based probabilistic framework.
• Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization — LLM-driven kernel discovery via a two-stage process and LOO-CRPS significantly improves high-dimensional Bayesian optimization.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.