Why force models to compute knowledge when they could just look it up?

2023-11-03 · Source: AIModels.fyi - Aimodels.substack.com · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, medium

Summary

A new paper, "Conditional Memory via Scalable Lookup," introduces Engram, a module that augments large language models (LLMs) with conditional memory, proposing a "second axis of sparsity" beyond traditional computational allocation. Engram integrates a static lookup table, indexed by token n-grams, allowing the model to instantly retrieve learned patterns and fuse them with ongoing computation via a learned gating scalar. This mechanism frees the transformer backbone's reasoning capacity, which typically wastes resources on factual retrieval. The research demonstrates a U-shaped tradeoff between memory and computation, revealing an optimal allocation where hybrid models outperform pure-computation baselines at equal parameters and FLOPs. Empirically, Engram improves knowledge-intensive tasks (MMLU by 3.4 points, CMMLU by 4.0) and, counterintuitively, reasoning tasks (BBH by 5.0, ARC-Challenge by 3.7, HumanEval by 3.0, MATH by 2.4). This is attributed to early layers focusing on abstraction, deepening the effective reasoning pipeline, and attention heads tracking longer-range relationships, boosting Multi-Query Needle in a Haystack from 84.2 to 97.0. The system's deterministic addressing enables efficient runtime prefetching, ensuring practical inference speed.

Key takeaway

For AI Architects designing next-generation LLMs, consider integrating conditional memory modules like Engram to optimize resource allocation. You can achieve superior performance by balancing static lookup memory with dynamic computation, especially for models scaling to 27B parameters or more. This approach not only boosts factual recall but also significantly enhances reasoning capabilities and long-context understanding by freeing up your model's core processing.

Key insights

Augmenting LLMs with conditional memory via lookup tables frees computational resources for enhanced reasoning.

Principles

• Optimal resource allocation between memory and computation follows a U-shaped curve.
• Delegating memorization deepens effective reasoning capacity in LLMs.
• Deterministic memory addressing enables efficient runtime prefetching.

Method

Engram fuses n-gram embeddings from a static lookup table into transformer hidden states using a learned gating scalar (0-1) that decides per-token whether to trust memory or computation.

In practice

• Integrate Engram modules into early transformer layers for maximal benefit.
• Design memory systems with deterministic addressing for practical inference speed.

Topics

• Large Language Models
• Conditional Memory
• Engram
• Sparsity Optimization
• Transformer Architecture
• Resource Allocation

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• AI 101: Conditional Memory and the Rise of Selective Intelligence — Selective intelligence allows models to choose memory access, improving efficiency and reasoning beyond traditional dense computation.
• Engram: How LLMs Finally Get Scalable Memory — Engram enhances LLM factual recall and reasoning by integrating a scalable, hash-based memory lookup system for direct knowledge retrieval.
• Learning When to Attend: Conditional Memory Access for Long-Context LLMs — Conditional attention mechanisms can significantly extend LLM context windows while improving efficiency.
• MILES: Modular Instruction Memory with Learnable Selection for Self-Improving LLM Reasoning — MILES improves LLM reasoning by dynamically expanding modular memory and using learnable selection for correctness-optimized composition.
• ReM-MoA: Reasoning Memory Sustains Mixture-of-Agents Scaling — Structured cross-layer reasoning memory and curated diversified routing are critical for scalable Mixture-of-Agents performance.
• LLM Inference and Optimization: Fundamentals, Bottlenecks, and Techniques — Optimizing LLM inference requires understanding distinct prefill and decode phases, leveraging KV caching, and applying advanced techniques.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AIModels.fyi - Aimodels.substack.com.