Beyond N-gram: Data-Aware X-GRAM Extraction for Efficient Embedding Parameter Scaling

2026-04-23 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

X-GRAM is a novel frequency-aware dynamic token-injection framework designed to improve the parameter efficiency and memory scaling of large token-indexed lookup tables, which often suffer from Zipfian under-training, heterogeneous demand, and "slot collapse." This framework uses hybrid hashing and alias mixing to compress the long tail of tokens while maintaining head capacity. It refines retrieved vectors using normalized SwiGLU ShortConv to extract diverse local n-gram features, integrating these signals into attention value streams and inter-layer residuals via depth-aware gating. This approach creates a memory-centric scaling axis that decouples model capacity from FLOPs. Evaluations on 0.73B and 1.15B scale models demonstrate that X-GRAM boosts average accuracy by up to 4.4 points over vanilla backbones and 3.2 points over strong retrieval baselines, even with 50% smaller tables.

Key takeaway

For AI Engineers developing large language models with token-indexed lookup tables, X-GRAM offers a practical paradigm to enhance parameter efficiency and memory scaling. By decoupling model capacity from FLOPs, you can achieve significant accuracy gains, up to 4.4 points, with substantially smaller embedding tables. Consider integrating X-GRAM's dynamic token injection and n-gram feature extraction to optimize your model's memory footprint and performance.

Key insights

X-GRAM improves embedding efficiency by dynamically managing token frequency and integrating refined n-gram features.

Principles

• Decouple capacity from compute via memory management.
• Address Zipfian distribution in embedding tables.
• Compress tail while preserving head capacity.

Method

X-GRAM uses hybrid hashing, alias mixing, normalized SwiGLU ShortConv for n-gram feature extraction, and depth-aware gating to integrate signals into attention and residuals.

In practice

• Use X-GRAM for memory-augmented architectures.
• Apply frequency-aware token injection.
• Integrate local n-gram features into attention.

Topics

• X-GRAM
• Embedding Parameter Scaling
• Token-indexed Lookup Tables
• Hybrid Hashing
• SwiGLU ShortConv

Code references

• Longyichen/X-gram

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models — Kronecker Embeddings drastically cut LLM input-side parameters while improving performance and spelling robustness.
• Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs — Tensorized Engram uses shared factors in CP form to efficiently represent n-gram embeddings, reducing parameters and improving performance.
• The Tale of Bloom Embeddings and Unseen Entities — Bloom embeddings in spaCy offer powerful, memory-efficient representation, outperforming traditional methods, especially for unseen entities.
• Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate — Maximal Update (\$μ\$P) improves LLM hyperparameter transfer by maximizing the embedding layer's learning rate, a bottleneck in standard parameterization.
• EntmaxKV: Support-Aware Decoding for Entmax Attention — EntmaxKV enables efficient, exact sparse decoding for long-context models by exploiting Entmax's zero-producing property.
• N-Grams and Markov Assumptions: The First Predictive Models of Language — N-grams were the first predictive language models, using statistical counts over short word sequences to anticipate the next word.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.