Tabular LLMs: An Introduction to the Foundation Models That Predict Your Spreadsheet

2026-07-24 · Source: Towards Data Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Software Development & Engineering · Depth: Advanced, extended

Summary

A new class of "tabular LLMs," specifically pretrained transformers, is outperforming traditional gradient-boosted trees (GBDTs) on tabular data prediction, as evidenced by the TabArena benchmark (snapshot 2026–07–15). The article independently verifies TabICLv2, a 28M-parameter open-weight model, by re-running its Lite protocol on 51 datasets on a single AWS A10G for \$2. This audit confirmed TabICLv2's Elo 1559 score (official 1575), with a median per-task difference of 0.08%. TabICLv2, which uses in-context learning and synthetic pretraining, achieves 158 Elo higher than the best tuned-and-ensembled GBDT, with significantly faster fit times (4 seconds per 1,000 rows vs. 100x longer for GBDTs). While TabFM (Google Research) tops the board at 123 Elo above TabICLv2, its weights are non-commercial and lack a paper. TabICLv2 excels on tables under 100 features and modest row counts, but GBDTs still win on very wide tables or those with high-cardinality categoricals.

Key takeaway

For Machine Learning Engineers evaluating tabular prediction models, if your datasets have under 100 features and modest row counts, you should prioritize TabICLv2. Its open weights, synthetic pretraining, and zero-shot performance offer superior accuracy and significantly faster "fit" times compared to tuned GBDTs. For very wide tables or those with wild high-cardinality categoricals, continue to use tuned GBDTs. Consider routing between foundation models for improved accuracy, but avoid validation-based routing between FMs and GBDTs.

Key insights

Tabular foundation models, like TabICLv2, now lead benchmarks for zero-shot tabular prediction, surpassing tuned GBDTs.

Principles

• Synthetic pretraining ensures benchmark contamination is ruled out.
• In-context learning enables zero-shot prediction without gradient training.
• Elo ratings are pool-dependent; use per-dataset metrics for independence.

Method

TabICLv2 uses three stacked transformers: one for column embedding, one for row vector collapse via CLS tokens, and a third for in-context prediction by attending over labeled rows.

In practice

• Use `pip install tabicl` for TabICLv2.
• Fit `TabICLClassifier()` on `X_train, y_train` in seconds.
• Predict `proba = clf.predict_proba(X_test)` in one pass.

Topics

• Tabular Foundation Models
• TabICLv2
• TabArena Benchmark
• Gradient-Boosted Trees
• In-Context Learning
• Zero-Shot Prediction
• Synthetic Data Pretraining

Code references

• soda-inria/tabicl

Best for: Machine Learning Engineer, AI Scientist, Data Scientist

Related on AIssential

• Introducing TabFM: A zero-shot foundation model for tabular data — TabFM applies zero-shot in-context learning to tabular data, eliminating manual tuning and feature engineering for classification and regression.
• Towards Universal Tabular Embeddings: A Benchmark Across Data Tasks — TEmBed benchmarks tabular embeddings across four representation levels to guide model selection.
• [R] Analysis of 350+ ML competitions in 2025 — ML competition trends show shifts towards foundation models and increased compute, while GBDTs and PyTorch maintain strong positions.
• TabH2O: A Unified Foundation Model for Tabular Prediction — TabH2O unifies tabular classification and regression via in-context learning, improving training stability and noise robustness.
• Large Tabular Models Excel Where LLMs Fail — Large Tabular Models (LTMs) are purpose-built foundation models that excel at analyzing structured data where LLMs fail.
• Attention Is Not What You Think! — In-context learning for tabular data is row attention, a mechanism Random Forests have employed for 20 years.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.