Kernel Regression with Tensor Trains and Hadamard Overparameterization

2026-07-22 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

Kernel regression with tensor trains and Hadamard overparameterization (KReTTaH) is a novel, training-data-free, and interpretable framework for multi-way data imputation. It reformulates imputation as regression in reproducing kernel Hilbert spaces, constraining tensor regression coefficients to fixed-rank tensor-train (TT) manifolds and structuring them with Hadamard overparameterization to promote sparsity and representational efficiency. KReTTaH jointly optimizes TT coefficient tensors and kernel covariance matrices on Riemannian product-manifolds, automating kernel-hyperparameter selection without costly cross-validation. Numerical tests on high-dimensional 4D-fMRI data and dynamic graph edge flow recovery demonstrate KReTTaH's superior modeling accuracy compared to existing tensor-, Bayesian-, and neural-network-based baselines. It also exhibits advantageous computational costs, often outperforming or matching competitors in speed, even with hyperparameter tuning.

Key takeaway

For AI Scientists or Machine Learning Engineers tackling multi-way data imputation, KReTTaH provides a powerful, training-data-free framework. You should consider this method for applications like fMRI reconstruction or dynamic graph flow imputation, especially when interpretability and computational efficiency are paramount. Its automated kernel hyperparameter selection and sparsity-promoting Hadamard overparameterization can streamline your workflow, delivering superior accuracy and competitive runtimes compared to traditional tensor or neural network approaches.

Key insights

KReTTaH integrates kernel regression, tensor trains, and Hadamard overparameterization via Riemannian optimization for efficient, interpretable multi-way data imputation.

Principles

• Fixed-rank TT manifolds enable dimensionality reduction and rich geometric structure.
• Hadamard overparameterization combined with smooth regularization promotes sparsity.
• Joint Riemannian optimization of coefficients and kernel matrices automates hyperparameter tuning.

Method

KReTTaH solves an inverse problem using Riemannian gradient descent with line search on a Cartesian-product Riemannian manifold, jointly updating all factors.

In practice

• Apply KReTTaH for missing value recovery in high-dimensional fMRI data.
• Utilize KReTTaH for imputing time-varying edge flows in complex networks.
• Consider multiple Gaussian kernels with learnable covariance matrices for improved performance.

Topics

• Kernel Regression
• Tensor Train Decomposition
• Hadamard Overparameterization
• Riemannian Optimization
• Multi-way Data Imputation
• fMRI Reconstruction

Code references

• bstabler/TransportationNetworks

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• From Chosen to Learned Kernels — Kernel geometry, critical for model performance, must be learned from data, not manually chosen, while addressing identifiability and leakage.
• Variational Low-rank Tensor Decomposition for Multisubject Spatiotemporal Data Analysis — Combining structured priors with unrolled optimization in variational tensor decomposition improves spatiotemporal data analysis.
• K-ABENA: K-Adaptive Backpropagation with Error-based N-exclusion Algorithm : (Compensated Loss-Based Sample Exclusion with Unbiased Gradient Estimation) — K-ABENA selectively excludes low-loss samples from backpropagation, achieving significant compute savings with unbiased gradient estimation and strong convergence.
• Generalized Least Squares Kernelized Tensor Factorization — GLSKF additively integrates global low-rank factorization and local correlated residuals for scalable, accurate multidimensional tensor completion.
• Global Fr{\'{e}}chet Manifold Learning for Random Objects, With Application to Low-Dimensional Wasserstein Representations of Distributional Data — Fréchet manifold learning creates low-dimensional Euclidean representations of metric data, enabling mapping back to the original space.
• A Smooth Alternative to the Boosted Tree — MS-SKM provides a smooth, interpretable kernel-based alternative to boosted trees for tabular data with smooth underlying structures.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.