Deep regression learning from dependent observations with minimum error entropy principle

2026-03-13 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

This paper introduces a deep regression learning approach for nonparametric regression using strongly mixing observations, a scenario where data points are dependent rather than independent. The method employs deep neural networks (DNNs) combined with the minimum error entropy (MEE) principle, which considers all moments of the error variable, offering robustness against non-Gaussian and heavy-tailed noise, unlike traditional $L_2$ (least squares) loss functions. Two specific estimators are analyzed: the non-penalized deep neural network (NPDNN) and the sparse-penalized deep neural network (SPDNN) predictors. The authors establish upper bounds for the expected excess risk of both estimators over Hölder and composition Hölder function classes. For models with Gaussian error, these MEE-based estimators achieve minimax optimal convergence rates, matching existing lower bounds up to a logarithmic factor. The study highlights that while the error density is assumed known, extending the work to unknown error densities remains a challenge.

Key takeaway

For AI Researchers and Research Scientists working on nonparametric regression with dependent data, this work demonstrates that MEE-based deep neural networks offer a robust alternative to $L_2$ loss, particularly for non-Gaussian or heavy-tailed noise. You should consider implementing NPDNN or SPDNN estimators, as they achieve minimax optimal convergence rates for strongly mixing observations. However, be aware that the current theoretical framework assumes a known error density, which may require further research for practical applications with unknown error distributions.

Key insights

MEE-based deep neural networks achieve minimax optimal rates for nonparametric regression with dependent data.

Principles

• MEE criteria account for all error moments, enhancing robustness.
• Strongly mixing data can achieve optimal convergence rates.
• Sparsity regularization improves DNN performance.

Method

The approach minimizes Shannon's entropy of the error using deep neural networks. It defines NPDNN and SPDNN estimators, establishing excess risk bounds over Hölder and composition Hölder function classes for strongly mixing observations.

In practice

• Use MEE for non-Gaussian or heavy-tailed noise.
• Consider SPDNN for improved robustness and sparsity.
• Apply to autoregressive processes with strong mixing.

Topics

• Deep Neural Networks
• Nonparametric Regression
• Minimum Error Entropy
• Strong Mixing
• Minimax Optimality

Best for: AI Researcher, AI Scientist, Research Scientist

Related on AIssential

• Generalized nonparametric regression in reproducing kernel Hilbert spaces: Consistency and rates of convergence — A new theory for RKHS M-estimation provides sharp convergence rates and explains how these estimators avoid the curse of dimensionality.
• Symmetrization of Loss Functions for Robust Training of Neural Networks in the Presence of Noisy Labels — Symmetrizing multi-class loss functions, especially cross-entropy, yields a unique convex unhinged loss robust to noisy labels.
• The Theory and Practice of Highly Scalable Gaussian Process Regression with Nearest Neighbours — NNGP/GPnn offer scalable, principled Gaussian process regression with minimax-optimal rates and hyperparameter robustness.
• Structured Nonparametric Variational Inference for Dependent Latent Modeling — SN-VI models complex latent variable dependencies using multivariate splines for accurate posterior approximation.
• Error Analysis for Deep ReLU Feedforward Density-Ratio Estimation with Bregman Divergence — BDD density-ratio estimators achieve minimax optimal error bounds and mitigate the curse of dimensionality.
• Deep numerical schemes for systems of Ergodic BSDEs with applications to regime-switching forward utilities — Neural network schemes effectively solve coupled ergodic BSDEs for regime-switching forward utilities.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.