In-Context Learning Demystified: The Kernel Machine Perspective

2026-01-11 · Source: Agus’s Substack · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, medium

Summary

In-context learning (ICL), often perceived as a mysterious emergent property in large language models, is fundamentally a process of "weighted voting based on similarity," akin to kernel machines. This approach, which requires no gradient updates or parameter tuning at inference, directly parallels how Transformers operate. The core mechanism involves computing similarity weights between a query point and context examples using a kernel function, normalizing these into attention weights via softmax, and then aggregating labels through weighted voting. This method is demonstrated using a simple Radial Basis Function (RBF) kernel in Python, showing how predictions adapt to context examples without retraining. This perspective demystifies attention and foundation models, presenting them as sophisticated kernel machines with learned similarity functions.

Key takeaway

For AI Engineers and Machine Learning Engineers seeking to understand and implement in-context learning, recognize that it is a form of kernel smoothing. You should focus on designing effective similarity functions, whether explicit kernels or learned embeddings, to build ICL systems. This understanding allows you to create adaptable models without extensive retraining, making complex "emergent" behaviors interpretable and reproducible.

Key insights

In-context learning is fundamentally kernel smoothing with optionally learned similarity functions.

Principles

• Attention IS kernel smoothing.
• ICL doesn't require training.
• Similarity-based voting drives ICL.

Method

Predict by computing similarity k(x, xᵢ) between query x and context xᵢ, normalizing to weights wᵢ(x) = k(x,xᵢ)/∑k(x,xⱼ), and aggregating labels: y^(x)=∑wᵢ(x)⋅yᵢ.

In practice

• Use RBF, cosine, or polynomial kernels.
• Implement learned embeddings for kernels.
• Apply Scott's rule for bandwidth selection.

Topics

• In-Context Learning
• Kernel Machines
• Attention Mechanisms
• Transformer Models
• Similarity Functions

Code references

• asudjianto-xml/substack

Best for: AI Engineer, Machine Learning Engineer, AI Scientist

Related on AIssential

• The Learned Kernel: How Transformers Actually Do In-Context Learning — Transformers excel at in-context learning by dynamically learning optimal similarity functions through their attention mechanisms.
• In-Context Learning for the Imputation of Public Opinion Data with Large Language Models — In-context learning with LLMs significantly improves missing public opinion data imputation over traditional statistical methods.
• Google's Warning: ICL Context is Inert — LLMs encode complex topologies internally but cannot functionally utilize these in-context learned representations for multi-step reasoning.
• Agentic Framework for Deep Learning workload migration via In-Context Learning — Combining ICL with oracle-driven self-debugging automates reliable deep learning model migration between frameworks.
• Learning to Select Visual In-Context Demonstrations — Learned, diversity-aware demonstration selection significantly improves MLLM performance on objective visual regression tasks.
• Categorical Prior Lock-in: Why In-Context Learning Fails for Structured Data — In-context learning struggles with categorical distribution shifts due to prior lock-in, failing to reproduce rare classes.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Agus’s Substack.