Token Geometry

2026-07-01 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Ember is a lightweight optimizer designed for language models, specifically targeting the embedding table and LM-head matrices. It exploits the distinct gradient geometry of these interfaces, which act as read/write mechanisms between continuous programs and discrete symbols. Ember significantly reduces VRAM usage to O(V + D) compared to Adam's O(2VD), eliminating the need to shard token table optimizer states and requiring only kilobytes of optimizer state. Empirical evidence demonstrates its effective scalability across various batch sizes and parameter counts. The research also reveals that token optimization trajectories can be described by a simple 1D ray, challenging the common perception of neural network parameters navigating a heavily nonconvex landscape. This provides a principled perspective on the narrow range of optimizers sufficient for Transformer training. A distributed Ember implementation, compatible with ZeRO/FSDP setups, has been open-sourced.

Key takeaway

For Machine Learning Engineers optimizing large language models, Ember offers a critical solution for VRAM efficiency. You should consider integrating Ember, which reduces optimizer state VRAM for embedding and LM-head matrices from O(2VD) to O(V + D), especially when working with memory-constrained environments or very large vocabularies. This allows you to scale models more effectively and potentially avoid complex sharding strategies for token tables, streamlining your training workflows.

Key insights

Ember optimizes language model embedding and LM-head matrices by leveraging their distinct gradient geometry for improved efficiency and performance.

Principles

• Embedding and LM-head gradients have distinct geometry.
• Token optimization trajectories can be 1D rays.
• Simple optimizers may suffice for Transformer training.

Method

Ember utilizes O(V + D) VRAM by exploiting the distinct gradient geometry of embedding and LM-head matrices, avoiding Adam's O(2VD) and the need for token table optimizer state sharding.

In practice

• Integrate Ember with existing ZeRO/FSDP setups.
• Reduce VRAM for embedding/LM-head optimization.

Topics

• Ember Optimizer
• Language Models
• VRAM Optimization
• Transformer Training
• Gradient Geometry
• Distributed Training

Code references

• katop1234/ember

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Building a Small Language Model (1) — Understanding Transformer — LLMs predict the next token by processing numerical representations of text through a multi-stage Transformer architecture.
• OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning — OccamToken uses register-anchored relative evidence testing for training-free, budget-adaptive VLM token pruning, improving efficiency.
• Compressing Sequences in the Latent Embedding Space: $K$-Token Merging for Large Language Models — K-Token Merging compresses LLM inputs in the latent embedding space for efficiency.
• DeepSeek-V3 Model: Theory, Config, and Rotary Positional Embeddings — DeepSeek-V3 integrates MLA, MoE, MTP, and RoPE to enhance LLM efficiency, capacity, and long-range understanding.
• Tokenisation via Convex Relaxations — A new tokenization algorithm, ConvexTok, uses convex optimization for globally optimal vocabulary construction.
• Token-Operations-Oriented Inference Optimization Techniques for Large Models — Token-oriented inference optimization requires a four-layer architecture to balance quality, cost, and stability for large model services.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.