Language Models Learn Universal Representations of Numbers and Here's Why You Should Care

2026-04-23 · Source: cs.NE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

This research investigates how Large Language Models (LLMs) process and represent numerical information, addressing the conflict between their accurate internal number embeddings and their propensity for numerical output errors. The study finds that diverse LLMs, including OLMo 2, Llama 3, and Phi 4, converge to systematic, highly accurate, and universal sinusoidal representations of numbers across their hidden states and input contexts. These representations are consistent across layers, primarily maintained by residual streams, though input/output embeddings are more distributed than the sparser internal representations. The authors developed universal sinusoidal probes that can accurately extract numeric information and attribute up to 94% of arithmetic reasoning errors in models like Llama 3.2 3B to specific internal layers, particularly in division. The work also shows that multi-token numbers are systematically superposed, with high accuracy for up to three tokens (up to 10^9).

Key takeaway

Research Scientists developing or fine-tuning LLMs should focus on the internal sinusoidal representations of numbers. Understanding these universal representations allows for the creation of more accurate probing techniques, which can pinpoint specific layers responsible for numerical errors. This insight enables targeted architectural adjustments to improve arithmetic reasoning and overall numerical accuracy, especially for multi-token numbers, potentially reducing errors by 27-64% in operations like division.

Key insights

LLMs use universal, systematic sinusoidal representations for numbers, enabling precise error tracing to specific internal layers.

Principles

• Number representations are sinusoidal and universal across LLMs.
• Residual streams maintain consistency of numeric representations.
• Natural-language probes generalize better than synthetic probes.

Method

The study uses Representational Similarity Analysis (RSA) and Fourier decompositions to quantify embedding similarity. It employs a sinusoidal probe, defined as $f_{\sin}(\mathbf{x}) =(\mathbf{W}_{\mathrm{out}}\mathbf{S})^{T}(\mathbf{W}_{\mathrm{in}}\mathbf{x})$, to decode internal representations and track error origins.

In practice

• Train probes using natural-language contexts for better generalization.
• Identify specific layers causing arithmetic errors in LLMs.
• Consider architectural refinements based on error-prone layers.

Topics

• Language Models
• Numeric Representations
• Sinusoidal Probes
• Model Interpretability
• Error Tracking

Code references

• prompteus/numllama

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Inside The Black Box: Now Read the Mind of the AI — LLMs represent concepts as curved manifolds in high-dimensional spaces, requiring advanced mathematical tools for interpretability.
• Towards Intrinsic Interpretability of Large Language Models:A Survey of Design Principles and Architectures — Intrinsic interpretability builds transparency directly into LLM architectures to enhance trustworthiness and safety.
• Testing LLM Arithmetic Reasoning Generalization with Automatic Numeric-Remapping Attacks — LLMs exhibit significant arithmetic reasoning fragility even with small, schema-preserving numeric changes, particularly on complex datasets like GSM8K.
• LLM Parameters for Math Across Languages: Shared or Separate? — LLM mathematical reasoning involves partially shared and partially language-specific parameters, with English dominating parameter sets.
• Why LLMs Get Math Wrong — Large Language Models (LLMs) often make mistakes in complex math problems and misinterpret tables because they process all input as flat text token sequences, treating math as patterns to predict rath…
• Disentangling Direction and Magnitude in Transformer Representations: A Double Dissociation Through L2-Matched Perturbation Analysis — Direction and magnitude of Transformer hidden states have distinct computational roles in LayerNorm-based architectures.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.NE updates on arXiv.org.