On the Robustness of Kernel Goodness-of-Fit Tests

2024-12-31 · Source: JMLR · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A new study by Xing Liu and François-Xavier Briol, published in 2025, addresses the robustness of kernel goodness-of-fit (GoF) tests, which are often criticized for rejecting models as sample sizes increase, given that "all models are wrong." The research demonstrates that current kernel GoF tests lack both qualitative and quantitative robustness under common definitions. Furthermore, the authors show that robustification methods like tilted kernels, effective in parameter estimation, do not adequately ensure robustness in the testing context. To overcome these limitations, the paper introduces the first robust kernel GoF test, which leverages kernel Stein discrepancy (KSD) balls. This novel framework is capable of encompassing various established perturbation models, including Huber's contamination and density-band models, resolving a long-standing open problem in the field.

Key takeaway

For AI Researchers and Statisticians evaluating probabilistic models, recognize that standard kernel goodness-of-fit tests are not robust and will likely reject even "good enough" models with sufficient data. You should consider adopting the new robust kernel GoF test based on kernel Stein discrepancy (KSD) balls to more accurately assess if your model is a mild perturbation of the true data-generating process, ensuring practical relevance in your model validation efforts.

Key insights

Existing kernel goodness-of-fit tests lack robustness, necessitating a new framework for practical model validation.

Principles

• "All models are wrong" implies GoF tests always reject with large samples.
• Robustness is critical for practical GoF testing.
• Tilted kernels are insufficient for GoF test robustness.

Method

The proposed robust kernel goodness-of-fit test utilizes kernel Stein discrepancy (KSD) balls to handle various perturbation models, including Huber's contamination and density-band models.

In practice

• Apply KSD balls for robust GoF testing.
• Consider Huber's contamination for perturbation models.

Topics

• Goodness-of-Fit Testing
• Kernel Methods
• Robustness
• Kernel Stein Discrepancy
• Perturbation Models

Code references

• XingLLiu/robust-kernel-test

Best for: AI Researcher, AI Scientist, Research Scientist

Related on AIssential

• Robust Local Polynomial Regression with Similarity Kernels — RSKLPR enhances LPR robustness by using similarity kernels that weight data based on both predictor and response variables.
• Computational-Statistical Trade-off in Kernel Two-Sample Testing with Random Fourier Features — Approximated MMD tests with Random Fourier Features can match full MMD power at sub-quadratic time with careful feature selection.
• Stein’s method, learning and inference -or- how to really monitor convergence and thin chains — Stein's method, leveraging scores and operators, offers a robust approach for convergence monitoring and advanced probabilistic inference.
• Zero-Shot Test-Time Canonicalization using Out-of-Distribution Scoring — Reframing test-time canonicalization as OOD detection improves vision model robustness to affine transformations without retraining.
• The Matching Principle: A Geometric Theory of Loss Functions for Nuisance-Robust Representation Learning — Robustness techniques converge on estimating and regularizing encoder Jacobians against label-preserving deployment nuisance covariance.
• The Kernel as the Normal Form — Most machine learning methods are variations of a kernel machine, with geometry (the kernel) as the primary choice.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by JMLR.