Universality of High-Dimensional Logistic Regression and a Novel CGMT under Dependence with Applications to Data Augmentation

2025-02-10 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

The paper "Universality of High-Dimensional Logistic Regression and a Novel CGMT under Dependence with Applications to Data Augmentation" by Matthew Esmaili Mallory, Kevin Han Huang, and Morgane Austern, submitted on 10 Feb 2025 (v1) and last revised 7 Jul 2026 (v4), addresses limitations in high-dimensional model risk characterization. Previous research, relying on Gaussian universality and the convex Gaussian min-max theorem (CGMT), assumed independent random vectors, restricting applicability. This work generalizes these foundational results to dependent settings. Specifically, it proves Gaussian universality holds for high-dimensional logistic regression under block dependence, m-dependence, and certain mixing cases. It also establishes a novel CGMT framework that accommodates correlation across both covariates and observations. These advancements are then used to determine the impact of data augmentation, a common deep learning practice, on asymptotic risk. The paper was published in the Proceedings of Thirty Eighth Conference on Learning Theory, PMLR 291:1799-1918, 2025.

Key takeaway

For research scientists modeling high-dimensional data with dependencies, this work provides crucial theoretical foundations. You can now rigorously analyze logistic regression models where data exhibits block dependence, m-dependence, or mixing, moving beyond the restrictive independent data assumption. This enables more accurate asymptotic risk predictions for models trained on real-world, correlated datasets, including those employing data augmentation in deep learning. Consider applying this generalized CGMT framework to validate your model's robustness.

Key insights

The paper extends high-dimensional model risk characterization to dependent data, proving Gaussian universality and introducing a novel CGMT framework.

Principles

• Gaussian universality extends to dependent data.
• CGMT can accommodate correlated data.
• Data augmentation impacts asymptotic risk.

Method

The authors generalize Gaussian universality for logistic regression under block, m-dependence, and mixing. They establish a new CGMT framework for correlated covariates and observations.

In practice

• Analyze logistic regression with dependent features.
• Evaluate data augmentation's risk impact.

Topics

• High-Dimensional Logistic Regression
• Gaussian Universality
• Convex Gaussian Min-Max Theorem
• Dependent Data
• Data Augmentation
• Asymptotic Risk

Best for: AI Scientist, Research Scientist

Related on AIssential

• Characterization of Gaussian Universality Breakdown in High-Dimensional Empirical Risk Minimization — Gaussian universality in high-dimensional ERM is not universal; its breakdown can be precisely characterized for non-Gaussian data.
• Covariate-dependent Hierarchical Dirichlet Processes — CD-HDP integrates covariates into hierarchical Bayesian nonparametrics for flexible density estimation and cluster identification.
• InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate — InfoAtlas reformulates mutual information estimation as a direct inference task, enabling zero-shot, real-time statistical dependence analysis.
• TrIM: Transformed Iterative Mondrian Forests for Gradient-based Dimension Reduction and High-Dimensional Regression — TrIM combines Mondrian forests with iterative gradient-based transformations for efficient, data-adaptive dimension reduction in high-dimensional regression.
• Harmonizing Multi-Objective LLM Unlearning via Unified Domain Representation and Bidirectional Logit Distillation — Multi-objective LLM unlearning requires harmonizing data representation and employing bidirectional distillation for balanced efficacy and robustness.
• Learning with Simulators: No Regret in a Computationally Bounded World — Given a simulator for dependent data, classical VC dimension learning guarantees can be recovered, broadening the PAC model.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.