Scaling Laws for Classical Machine Learning on Tabular Data: A Benchmark Study

2026-07-23 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, quick

Summary

A large-scale replication study investigated scaling laws for classical machine learning models on tabular data, involving 127 students across 18 datasets and 6 model families (Boosting, Random Forest, SVM, Linear/Logistic, Ridge, Lasso). This yielded 11,536 training runs and 1,648 fitted power-law curves of the form error(N) = a N^(-b) + c. The study found power laws fit well (R^2 > 0.8 on 77.7% of cells), with tree ensembles dominating at full data. Five of six model families showed approximately shared exponents across datasets, predicting curves nearly as well as per-dataset exponents (R^2 gap < 0.011), though this was attributed to approximate predictive compressibility, not universality. Significant replicator-implementation variance was observed, with a mean CV(b) = 0.144 difference in fitted exponents even with "random_state=42" fixed, due to unconstrained preprocessing. Aggregated curves and a data-requirement table for N* to reach target error 0.15 are released.

Key takeaway

For Machine Learning Engineers estimating data requirements or predicting model performance on tabular data, you should recognize that while power laws (error(N) = a N^(-b) + c) generally apply, implementation details like preprocessing significantly influence the fitted exponents. When deploying models, account for this "replicator-implementation variance" by carefully standardizing your data handling protocols. You can use the released data-requirement table to guide your data acquisition targets for a desired error of 0.15, but validate these estimates against your specific pipeline.

Key insights

Classical ML scaling laws (error(N) = a N^(-b) + c) fit tabular data, but implementation variance is substantial.

Principles

• Power laws model classical ML performance.
• Model families share approximate scaling exponents.
• Preprocessing choices affect scaling law parameters.

Method

A distributed classroom-scale replication involved 127 students running a fixed protocol on 18 datasets with 6 model families, fitting error(N) = a N^(-b) + c curves.

In practice

• Estimate data needs using error(N) = a N^(-b) + c.
• Account for preprocessing in scaling predictions.
• Leverage family-level exponents for model selection.

Topics

• Scaling Laws
• Classical Machine Learning
• Tabular Data
• Model Performance
• Replication Study
• Preprocessing Variance

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Scaling Laws for Classical Machine Learning on Tabular Data: A Benchmark Study — Classical ML models on tabular data exhibit predictable power-law scaling, but implementation variance and irreducible error are critical.
• Scaling Laws, Carefully — Scaling laws predict deep learning loss based on model size, data, and compute, but optimal allocation depends critically on experimental scale and data availability.
• Tabular LLMs: An Introduction to the Foundation Models That Predict Your Spreadsheet — Tabular foundation models, like TabICLv2, now lead benchmarks for zero-shot tabular prediction, surpassing tuned GBDTs.
• Towards Engineering Scaling Laws with Pretraining Data Composition — Pretraining data composition can engineer neural scaling laws in particle physics, favoring data-rich regimes over larger models.
• Explaining Data Mixing Scaling Laws — A unified framework explains data mixing scaling laws through capacity competition and noise reduction.
• Learning From Examples Is Geometry Discovery Too — Tabular ICL models learn a Bayesian-optimal similarity function from synthetic data, acting as an advanced k-NN.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.