Scaling Laws for Classical Machine Learning on Tabular Data: A Benchmark Study

2026-07-27 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Research Methodology & Innovation · Depth: Expert, extended

Summary

A distributed classroom-scale replication study involving 127 students analyzed scaling laws for classical machine learning models on tabular data. Utilizing 18 datasets and 6 model families (Boosting, Random Forest, SVM, Linear/Logistic, Ridge, Lasso), the study generated 11,536 training runs and 1,648 power-law curves of the form error(N)=aN^{-b}+c. Key findings include that power laws fit well, with R^2 > 0.8 on 77.7% of cells, and tree ensembles dominate at full data (Boosting 50%, Random Forest 33%). The research also found approximate shared exponents within 5 of 6 model families, where a single family-level exponent predicted cross-dataset curves with an R^2 gap < 0.011, despite AIC favoring unconstrained fits. Furthermore, a significant replicator-implementation variance was observed, with a mean CV(b)=0.144 on fitted exponents, stemming from unconstrained protocol elements like preprocessing. The study releases aggregated curves, per-cell fits, and a data-requirement table for N* to reach a target error of 0.15.

Key takeaway

For Machine Learning Engineers designing or budgeting for tabular data projects, you should recognize that classical models exhibit predictable scaling laws, but implementation variance is significant. When estimating data needs, consult the provided N* requirement table and consider that if your model's irreducible error "c" is high, more data is unlikely to help; focus instead on feature engineering or model class selection. Your scaling exponent claims should account for a ~14% protocol-drift floor.

Key insights

Classical ML models on tabular data exhibit predictable power-law scaling, but implementation variance and irreducible error are critical.

Principles

• Classical ML error on tabular data follows aN^-b+c power laws.
• Model families exhibit approximate shared scaling exponents.
• Replicator-implementation variance affects scaling exponent reliability.

Method

A fixed protocol was executed by 127 students on 18 datasets and 6 models, fitting error(N) = aN^-b + c to nested training subsets. Cross-student variance of "b" was measured.

In practice

• Use the provided data-requirement table for N* to reach target error 0.15.
• Account for ~14% replicator-implementation variance in scaling exponent "b".
• If irreducible error "c" is high, focus on features or model class, not more data.

Topics

• Scaling Laws
• Tabular Machine Learning
• Model Performance Benchmarking
• Reproducibility
• Data Requirements Estimation
• Tree Ensembles

Code references

• dingkaihua/classical_scaling

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Student

Related on AIssential

• Scaling Laws for Classical Machine Learning on Tabular Data: A Benchmark Study — Classical ML scaling laws (error(N) = a N^(-b) + c) fit tabular data, but implementation variance is substantial.
• On the Smallness of the Large Language Models Scaling Exponents — LLM scaling exponents indicate an unsustainable energy regime, unmitigated by the "pedestal effect."
• Scaling Laws, Carefully — Scaling laws predict deep learning loss based on model size, data, and compute, but optimal allocation depends critically on experimental scale and data availability.
• Bayesian Wind Tunnels for Model Selection — Transformers perform quantitative Bayesian model selection, but their success depends on the "perceptual access" and "circuit depth" of the underlying structure.
• Towards Universal Tabular Embeddings: A Benchmark Across Data Tasks — TEmBed benchmarks tabular embeddings across four representation levels to guide model selection.
• Learning From Examples Is Geometry Discovery Too — Tabular ICL models learn a Bayesian-optimal similarity function from synthetic data, acting as an advanced k-NN.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.