Sequential Regression Learning with Randomized Algorithms

2026-04-20 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

This paper introduces "randomized SINDy," a sequential machine learning algorithm designed for dynamic, time-dependent data, moving beyond the traditional independent and identically distributed (iid) assumption. The algorithm employs a probabilistic approach, with its PAC (Probably Approximately Correct) learning property mathematically proven using functional analysis. It dynamically predicts outcomes by learning and updating a probability distribution of predictors, using gradient descent and a proximal algorithm to maintain a valid probability density. Inspired by the SINDy algorithm, it incorporates feature augmentation and Tikhonov regularization. For multivariate normal weights, the proximal step is omitted to focus on parameter estimation. The algorithm's effectiveness is demonstrated through extensive experiments in regression and binary classification using both simulated and real-world datasets, including U.S. unemployment rate forecasting and electricity price change prediction.

Key takeaway

For Machine Learning Engineers building models for dynamic, time-dependent data streams, randomized SINDy offers a robust approach that inherently provides confidence measures. You should consider implementing this algorithm, especially when traditional iid assumptions are violated, and explore its adaptive capabilities for real-time forecasting and classification tasks. Pay close attention to learning rate selection and initial parameter estimation, as these significantly influence convergence and model stability.

Key insights

Randomized SINDy offers a PAC-learnable sequential algorithm for dynamic, time-dependent data using probabilistic predictor distributions.

Principles

• Sequential learning is crucial for dynamic, high-volume data.
• Probabilistic predictor distributions enable confidence measures.
• Proximal algorithms ensure valid probability density updates.

Method

The method involves dynamically updating a probability distribution of predictors via gradient descent and a proximal algorithm, which acts as a projection operator, to maintain a valid probability density over the hypothesis space.

In practice

• Apply Tikhonov regularization for ill-conditioned data.
• Use rolling window cross-validation for hyperparameter tuning.
• Monitor residual charts for early concept drift detection.

Topics

• Randomized SINDy
• Sequential Machine Learning
• PAC Learning Property
• Proximal Algorithms
• Tikhonov Regularization

Best for: AI Scientist, Research Scientist, Machine Learning Engineer

Related on AIssential

• Revisiting Active Sequential Prediction-Powered Mean Estimation — Optimal confidence width in active sequential prediction-powered mean estimation favors a high constant query probability.
• AdaGrad-Diff: A New Version of the Adaptive Gradient Algorithm — AdaGrad-Diff improves optimization robustness by adapting stepsizes based on gradient differences, not just gradient magnitudes.
• How Low Can You Go? Active Learning for Sparse Model Discovery in the Ultra-Low-Data Limit — An active learning strategy using E-SINDy efficiently discovers sparse dynamical models by prioritizing informative data samples in ultra-low data regimes.
• Bootstrapping-based Regularisation for Reducing Individual Prediction Instability in Clinical Risk Prediction Models — A novel regularisation method improves clinical prediction model stability without sacrificing interpretability, crucial for healthcare adoption.
• OrderDP: A Theoretically Guaranteed Lossless Dynamic Data Pruning Framework — OrderDP provides theoretically guaranteed, unbiased, and near-lossless data pruning for accelerated model training.
• Learning with Simulators: No Regret in a Computationally Bounded World — Given a simulator for dependent data, classical VC dimension learning guarantees can be recovered, broadening the PAC model.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.