TrIM: Transformed Iterative Mondrian Forests for Gradient-based Dimension Reduction and High-Dimensional Regression

2026-06-16 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

TrIM (Transformed Iterative Mondrian Forests) is a new, computationally efficient algorithm designed for gradient-based linear dimension reduction and high-dimensional regression. It addresses challenges in high-dimensional data by identifying a low-dimensional "relevant feature subspace" rather than relying on sparse feature selection. The algorithm operates iteratively: it first computes an initial Mondrian forest, then uses this to estimate the expected gradient outer product (EGOP) matrix. This EGOP estimate is then used to create a linear transformation of the input features, on which a new Mondrian forest (an "oblique Mondrian forest") is built. This process can be iterated to refine the estimator. The authors provide consistency guarantees and convergence rates for both the EGOP matrix estimation (e.g., ℤ[||Ĥₙ-H||]≲ n⁻³/⁴ᵈ⁺¹²) and the one-iteration TrIM algorithm, demonstrating improved accuracy over standard Mondrian forests in various simulated and real-world datasets, including an Ebola spread model.

Key takeaway

For Machine Learning Engineers working with high-dimensional datasets and seeking improved model accuracy and interpretability, you should consider implementing TrIM forests. This algorithm offers a robust approach to identify and leverage low-dimensional feature subspaces, moving beyond traditional sparse feature selection. By iteratively transforming input data based on gradient information, TrIM can enhance regression performance, especially in scenarios where the underlying function is a ridge function. Explore the provided GitHub repository for practical implementation.

Key insights

TrIM combines Mondrian forests with iterative gradient-based transformations for efficient, data-adaptive dimension reduction in high-dimensional regression.

Principles

• Dimension reduction enhances model explainability.
• EGOP matrix identifies relevant feature subspaces.
• Iterative refinement improves model performance.

Method

TrIM iteratively estimates the Expected Gradient Outer Product (EGOP) using an initial Mondrian forest, then applies this linear transformation to the data, and finally builds an oblique Mondrian forest on the transformed features.

In practice

• Apply TrIM for high-dimensional regression tasks.
• Use EGOP to identify critical feature combinations.
• Utilize Mondrian forests for online data processing.

Topics

• Mondrian Forests
• Dimension Reduction
• High-Dimensional Regression
• Expected Gradient Outer Product
• Feature Subspace Learning
• Iterative Algorithms

Code references

• Xieyangxinyu/TrIM

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Hinge Regression Tree: A Newton Method for Oblique Regression Tree Splitting — Hinge Regression Trees use a damped Newton method to learn powerful oblique splits, achieving universal approximation.
• How does feature learning reshape the function space? — Feature learning in two-layer neural networks deforms function space via a data-adaptive, target-dependent kernel.
• MinShap: A Modified Shapley Value Approach for Feature Selection — MinShap adapts Shapley values for robust feature selection by focusing on minimum marginal contributions across permutations.
• Selecting Samples on Graphs: A Unified Dataset Pruning Framework for Lossless Training Acceleration — A unified graph-based framework prunes datasets by modeling samples as nodes and relations as edges, solving a Maximum Weight Clique Problem.
• The Hidden Self-Attention Inside Gradient-Boosted Trees — GBDTs, kernel regression, and self-attention share a QKV operator structure, differing in where they place complexity.
• Multiple Additive Neural Networks for Structured and Unstructured Data — MANN enhances gradient boosting with shallow neural networks for diverse data types, improving accuracy and robustness.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.