Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs

2026-06-06 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

Tensorized Engram (TN-gram) is a novel memory module designed for large language models, addressing limitations of discrete token-level embeddings and existing n-gram memory approaches like Engram. Unlike methods that use separate hash tables for each n-gram order, TN-gram employs shared factors in the Canonical Polyadic (CP) form to represent tensorized n-gram embeddings. It learns shared token-position factors and order-absorption vectors, enabling nested n-grams to share underlying latent structures. Experiments show TN-gram matches or outperforms Engram-style n-gram modules while significantly reducing parameter count. This research was published on 2026-06-06.

Key takeaway

For AI Scientists and Machine Learning Engineers optimizing LLM memory and efficiency, consider Tensorized Engram (TN-gram) as an alternative to traditional n-gram embedding methods. You can achieve comparable or superior performance to Engram-style modules while significantly reducing the parameter footprint. Evaluate TN-gram's Canonical Polyadic (CP) form for more compact and effective multi-token pattern representation in your next model architecture.

Key insights

Tensorized Engram uses shared factors in CP form to efficiently represent n-gram embeddings, reducing parameters and improving performance.

Principles

• Shared factors improve n-gram embedding efficiency.
• Canonical Polyadic form enables compact memory.
• Order-absorption vectors encode n-gram orders.

Method

TN-gram represents tensorized n-gram embeddings via shared factors in Canonical Polyadic (CP) form, learning token-position factors and order-absorption vectors to encode various n-gram orders.

In practice

• Implement CP decomposition for n-gram memory.
• Integrate shared latent structures for multi-token patterns.
• Reduce LLM parameter count for n-gram modules.

Topics

• Large Language Models
• N-gram Embeddings
• Tensor Decomposition
• Canonical Polyadic (CP) Form
• Model Efficiency
• Memory Optimization

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Beyond N-gram: Data-Aware X-GRAM Extraction for Efficient Embedding Parameter Scaling — X-GRAM improves embedding efficiency by dynamically managing token frequency and integrating refined n-gram features.
• A Typed Tensor Language for Federated Learning — The language formalizes federated learning computations, enabling fixed-dimensional shared state communication for efficiency.
• EinSort: Sorting is All We Need for Tensorizing LLM — EinSort uses index ordering for adaptive tensorization to compress LLMs by finding low-rank structures.
• Your UnEmbedding Matrix is Secretly a Feature Lens for Text Embeddings — LLM unembedding matrices encode an "edge spectrum" subspace biasing embeddings towards frequent, uninformative tokens, which EmbedFilter removes.
• Uncovering Symmetry Transfer in Large Language Models via Layer-Peeled Optimization — LLM training implicitly transfers data symmetries into structured geometric configurations within model parameters.
• Prism: Symbolic Superoptimization of Tensor Programs — Prism superoptimizes tensor programs using symbolic graphs and a two-level search for significant speedups.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.