A nonparametric two-sample test using a parametric integral probability metric

2026-06-16 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A new nonparametric two-sample test, PReLU-TST, is introduced for detecting distributional differences between two independent samples without assuming a specific parametric distribution. This 45-page study proposes a test statistic, PReLU-IPM, based on an integral probability metric (IPM) that utilizes a specially designed parametric discriminator class with a single neural network node. The research establishes theoretical guarantees for PReLU-TST, including its consistency and asymptotical equivalence to existing nonparametric IPM-based tests under regularity conditions. Empirical evaluations across multiple simulated and real benchmark datasets demonstrate that PReLU-TST achieves higher power across various alternatives or performs comparably to competitor methods for finite samples. The work has been accepted for publication in Statistical Analysis and Data Mining.

Key takeaway

For Machine Learning Engineers or Data Scientists evaluating distributional differences between datasets, PReLU-TST offers a powerful, assumption-free alternative. You should consider integrating this nonparametric test, especially for finite samples where it shows higher power or comparable performance. This could improve the reliability of your model validation or data quality checks, providing stronger statistical evidence for sample divergence.

Key insights

PReLU-TST offers a robust nonparametric two-sample test with strong theoretical guarantees and superior finite-sample power.

Principles

• Nonparametric tests avoid distribution assumptions.
• IPMs can form the basis for test statistics.
• Discriminator class design impacts test power.

Method

The PReLU-TST procedure constructs a test statistic, PReLU-IPM, using an integral probability metric with a single-node neural network as its parametric discriminator class.

In practice

• Apply PReLU-TST for robust sample comparison.
• Evaluate PReLU-TST on new benchmark datasets.
• Consider single-node neural networks for discriminators.

Topics

• Nonparametric Testing
• Two-Sample Test
• Integral Probability Metrics
• Neural Networks
• Distributional Differences
• Machine Learning Statistics

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Data Scientist

Related on AIssential

• LOTTERY: Learning from Reference-Only Samples in Two-Sample Testing under Size Asymmetry — LOTTERY enables robust two-sample testing in few-shot, imbalanced settings by learning reference-dependent representations from abundant reference data.
• Beyond Expected Information Gain: Stable Bayesian Optimal Experimental Design with Integral Probability Metrics and Plug-and-Play Extensions — IPM-based BOED offers superior stability and computational efficiency over traditional KL-divergence methods, especially in high-dimensional settings.
• Hypothesis Testing Revisited: Normal, t, and Chi-Squared Distribution Tests — Hypothesis testing follows a consistent procedure, adapting test statistics and distributions based on data characteristics and the hypothesis type.
• Computational-Statistical Trade-off in Kernel Two-Sample Testing with Random Fourier Features — Approximated MMD tests with Random Fourier Features can match full MMD power at sub-quadratic time with careful feature selection.
• Minimax Optimal Two-Sample Testing under Local Differential Privacy — Minimax optimal two-sample tests under LDP are developed for multinomial and continuous data.
• Conditional anomaly detection with soft harmonic functions — A new non-parametric method uses soft harmonic functions to detect conditional anomalies and mislabeling.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.