The Geometry Hidden in Every Model

2026-01-11 · Source: Agus’s Substack · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, long

Summary

The first post in The Learned Kernel series argues that every supervised model's prediction is a weighted average of training labels, where the weights define a "geometry" representing similarity. This concept is demonstrated using ridge regression, regression trees, and k-nearest neighbors on California housing data, yielding different predictions like \$306k, \$370k, and \$321k for a single block. Each model employs a distinct weighting function: ridge uses a global, signed elliptical influence; trees use local, axis-aligned boxes; and k-NN uses local, isotropic balls. The article posits that learning is the discovery of this geometry, which it terms the "kernel," and emphasizes that this kernel should be learned from data rather than pre-chosen.

Key takeaway

For Machine Learning Engineers evaluating model choices, understand that different algorithms imply distinct "geometries" of similarity, not just competing predictions. You should analyze a model's underlying weight function, or "kernel," to grasp how it defines relevance and influence from training data. This deeper understanding helps you move beyond arbitrary parameter tuning (like k in k-NN) towards learning the optimal geometry directly from your data, leading to more trustworthy and accurate models.

Key insights

Learning in supervised models is the discovery of a geometric kernel that defines similarity and weights training labels.

Principles

• Learning is geometry discovery.
• Models differ by weight functions.
• Geometry should be learned, not chosen.

Method

Supervised models compute predictions as weighted sums of training labels, y^(x)=∑i=1nwi(x)yi, where wᵢ(x) is the influence of training case i on query x, effectively defining the model's geometric kernel.

In practice

• Compare models by their weights, not just predictions.
• Visualize model influence as geometric shapes.
• Use provided Python functions to compute model weights.

Topics

• Supervised Learning
• Model Geometry
• Kernel Methods
• Ridge Regression
• Regression Trees
• k-Nearest Neighbors

Code references

• asudjianto-xml/Learned-Kernel

Best for: AI Scientist, Machine Learning Engineer, Data Scientist

Related on AIssential

• Geometry Without Coordinates — Relational geometry, defined by kernels and graphs, often reveals structure that raw coordinate-based methods obscure.
• You Have Been Doing Geometry All Along — Data science workflows are fundamentally geometric, shaping how models perceive and learn structure.
• Learning the Geometry of Data: A Mathematical Review of Shape Space Analysis — Shape space analysis offers a unified geometric framework to extract patterns from complex data, outperforming traditional ML on subtle variations and sparse labels.
• Why Every Supervised Model You’ve Ever Trained Has a Geometric Blind Spot (And Why More Data Won’t… — Supervised models have an inherent geometric blind spot due to their objective function, causing structural failures.
• Shape, Symmetries, and Structure: The Changing Role of Mathematics in Machine Learning Research — Mathematics' role in ML is shifting from theoretical guarantees to post-hoc explanations and high-level architectural guidance, leveraging advanced pure math.
• The Geometry of Linear Equations and Linear Regression — Geometric understanding of linear algebra clarifies solutions and underpins linear regression.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Agus’s Substack.