You Can Fit a Million Nearly-Perpendicular Arrows in 768 Dimensions.

2026-06-23 · Source: AI on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Intermediate, quick

Summary

The article explains the fundamental reason behind the effectiveness of word embeddings: the ability of high-dimensional spaces to accommodate a vast number of "nearly-perpendicular" vectors. While human intuition, trained in three dimensions, limits truly perpendicular arrows to three, models operating in spaces like 768 dimensions can fit millions of arrows that are very close to right angles to each other. This counter-intuitive property allows models to represent far more distinct ideas than their dimensional count suggests. The author plans to demonstrate this concept through arithmetic, starting with small vector examples and gradually increasing dimensions (2, 4, 100, then 768), ultimately showing how this "absurd fact" underpins every word embedding.

Key takeaway

For Machine Learning Engineers designing or debugging embedding-based systems, understanding that high-dimensional spaces can accommodate millions of nearly-perpendicular vectors is crucial. Your intuition from 3D space will mislead you about the true capacity of embeddings. This insight clarifies why models can represent a vast array of concepts, informing decisions on embedding dimension selection and the interpretability of vector space operations.

Key insights

High-dimensional spaces enable millions of nearly-perpendicular vectors, explaining why embeddings effectively represent numerous distinct ideas.

Principles

• High-dimensional geometry defies 3D intuition.
• Embeddings leverage near-orthogonality for capacity.
• Vector perpendicularity increases with dimensions.

Method

The article outlines a method to compute and observe the perpendicularity of random arrows in increasing dimensions (2, 4, 100, 768) to demonstrate embedding capacity.

Topics

• High-Dimensional Geometry
• Word Embeddings
• Vector Orthogonality
• Embedding Capacity
• Machine Learning Theory

Best for: AI Scientist, Machine Learning Engineer, AI Student

Related on AIssential

• Vector Embeddings — Vector embeddings transform complex data into numerical vectors, enabling machine learning to process semantic relationships.
• The most beautiful formula not enough people understand — High-dimensional geometry reveals counterintuitive properties, with n-dimensional sphere volumes peaking then becoming infinitesimal.
• Structure Retention in Embedding Spaces as a Predictor of Benchmark Performance — Embedding space geometry, specifically local structure retention and linear separability, strongly predicts model performance.
• Learning to Recall with Transformers Beyond Orthogonal Embeddings — Transformer storage capacity scales multiplicatively with sample size, embedding dimension, and sequence length under random embeddings.
• The Geometry of Representational Failures in Vision Language Models — VLM multi-object failures stem from geometric interference between concept vectors in a shared latent space.
• Universality of Shallow and Deep Neural Networks on Non-Euclidean Spaces — TFNNs extend universal approximation to non-Euclidean spaces, even with width constraints, by leveraging topological properties.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI on Medium.