Row and Column Attention Are Kernel Smoothing

2026-01-11 · Source: Agus’s Substack · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

Row attention and column attention, often presented as novel primitives in tabular transformers, are algebraically equivalent to Nadaraya-Watson kernel smoothing. This analysis demonstrates that softmax attention functions as Nadaraya-Watson smoothing using an exponential dot-product kernel. Specifically, row attention performs kernel smoothing over sample points (rows), while column attention applies the same kernel smoothing over features within a row to construct enhanced representations. The process involves mapping points into query and key representations, calculating an exponential dot-product kernel K(x⋆,xi)=exp(q⋆⊤ki), and then normalizing these to form Nadaraya-Watson weights. A Python/NumPy example confirms that attention weights are identical to these normalized kernel values. The primary innovation of tabular transformers is not the attention mechanism itself, but rather the learning and composition of the kernel, allowing the model to adaptively define the predictive geometry across tasks.

Key takeaway

For Machine Learning Engineers designing tabular prediction models, understand that transformer attention is a learned kernel smoothing technique. Evaluate its adaptive kernel against traditional methods like RBF kernels or tree-based models. Do not treat it as a fundamentally new primitive. Recognize that row attention smooths over samples and column attention over features, both leveraging Nadaraya-Watson principles. This perspective can inform your architectural choices and performance comparisons for tabular data.

Key insights

Softmax attention in tabular transformers is mathematically equivalent to Nadaraya-Watson kernel smoothing with a learned, composed exponential dot-product kernel.

Principles

• Attention is kernel smoothing, not a new primitive.
• Kernel geometry can be learned and composed.
• 1/√d acts as the anisotropic bandwidth in attention.

Method

Map data into query and key representations, compute an exponential dot-product kernel, normalize it via softmax for weights, then apply these weights in a Nadaraya-Watson average.

In practice

• Evaluate learned kernels against RBF or tree kernels.
• Consider attention as an adaptive nonparametric regressor.
• Use 1/√d as a fixed global scale for attention bandwidth.

Topics

• Tabular Transformers
• Kernel Smoothing
• Nadaraya-Watson
• Row Attention
• Column Attention
• Nonparametric Regression

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Data Scientist

Related on AIssential

• Attention Is Not What You Think! — In-context learning for tabular data is row attention, a mechanism Random Forests have employed for 20 years.
• The Kernel as the Normal Form — Most machine learning methods are variations of a kernel machine, with geometry (the kernel) as the primary choice.
• [D] A mathematical proof from an anonymous Korean forum: The essence of Attention is fundamentally a d^2 problem, not n^2. (PDF included) — An anonymous paper, "The d^2 Pullback Theorem," claims the intrinsic optimization landscape of the Attention mechanism is fundamentally d^2-dimensional, not n^2, arguing that the O(n^2) bottleneck is…
• A Histogram Without the Bin Edges — KDE provides a bin-free, smooth data density estimate by summing individual kernel "bumps" over each data point.
• Attention-based PCA — Attention mechanisms inherently perform PCA-like computations, aligning with principal eigenvectors under unsupervised training.
• Multiscale POD of Transformer Attention Fields: Scale-Selective Analysis via Morlet Scalogram — Scale-selective POD reveals layer-dependent attention scale organization in Transformers without architectural changes or linguistic annotations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Agus’s Substack.