Evidence Slopes and Effective Dimension in Singular Linear Models

2026-01-06 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

This research investigates the limitations of standard Bayesian model selection approximations, Laplace's method and BIC, in singular linear models. These approximations incorrectly assume the effective model dimension equals the number of parameters ($d$), whereas singular learning theory posits the real log canonical threshold (RLCT) $\lambda$ as the true effective dimension, which can be smaller in overparameterized or low-rank models. The study focuses on linear-Gaussian rank models, where RLCT and exact marginal likelihood are analytically tractable. It demonstrates that Laplace/BIC errors grow linearly with $(d/2-\lambda)\log n$, while an RLCT-aware correction accurately recovers the correct slope. The findings also show that RLCT and evidence are invariant to overcomplete reparameterizations with the same data subspace, unlike BIC, and propose interpreting the evidence versus $\log n$ slope as a practical RLCT estimator.

Key takeaway

For research scientists working with overparameterized or low-rank models, you should be aware that traditional Laplace/BIC approximations can significantly misestimate model complexity. Consider adopting RLCT-aware corrections, especially when dealing with singular models where the intrinsic rank is less than the ambient parameter dimension, to ensure accurate Bayesian evidence calculations and avoid biased model comparisons. This approach will provide more reliable insights into your model's true complexity.

Key insights

Singular models require RLCT for accurate Bayesian evidence, as Laplace/BIC over-penalize by $(d/2-\lambda)\log n$.

Principles

• Effective dimension is RLCT, not parameter count.
• RLCT-based penalties are representation-invariant.
• Laplace/BIC errors grow with $(d-r)\log n/2$ in singular models.

Method

Estimate RLCT by observing the slope of evidence versus $\log n$ in linear-Gaussian models, particularly for rank-deficient regression and linear dictionary models.

In practice

• Use RLCT-aware corrections for singular models.
• Avoid BIC for overcomplete representations.
• Analyze evidence slopes to infer effective dimension.

Topics

• Singular Learning Theory
• Real Log Canonical Threshold
• Bayesian Model Selection
• Overparameterized Models
• Evidence Slopes

Best for: Research Scientist, AI Researcher, AI Scientist, Machine Learning Engineer

Related on AIssential

• Occam's Razor is Only as Sharp as Your ELBO — ELBO's model selection behavior depends critically on the approximate posterior's covariance structure.
• Efficient Solvers for SLOPE in R, Python, Julia, and C++ — The new SLOPE packages offer superior performance and flexibility for regularized GLM estimation across multiple programming languages.
• Decorrelated Local Linear Estimator: Inference for Non-linear Effects in High-dimensional Additive Models — A new decorrelated local linear estimator enables robust inference for non-linear effects in high-dimensional additive models.
• Information Gap and Feasibility-Aware Inference in Binomial Logistic Mixtures — In binomial logistic mixtures, detecting components doesn't guarantee label recovery, an intrinsic information gap addressable by feasibility-aware inference.
• Top 20 Bayesian Regression Interview Questions and Answers (Part 1 of 2) — The Laplace prior promotes sparsity in Bayesian regression, improving interpretability and reducing overfitting when few features are important.
• Calibrating Uncertainty for Zero-Shot Adversarial CLIP — Adversarial attacks on CLIP suppress uncertainty, leading to over-confident misclassifications; calibrating this is crucial for reliability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.