LMEB: Long-horizon Memory Embedding Benchmark

2026-03-10 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, long

Summary

The Long-horizon Memory Embedding Benchmark (LMEB) is introduced as a comprehensive framework to evaluate embedding models for complex, long-horizon memory retrieval tasks, a capability underexplored by existing benchmarks like MTEB. LMEB spans 22 datasets and 193 zero-shot retrieval tasks across four distinct memory types: episodic, dialogue, semantic, and procedural, incorporating both AI-generated and human-annotated data. Evaluating 15 widely used embedding models, ranging from hundreds of millions to ten billion parameters, LMEB reveals that it offers a reasonable level of difficulty, with the top model achieving a Mean (Dataset) score of 61.41 on N@10. Crucially, the benchmark demonstrates that larger models do not consistently outperform smaller ones, and that LMEB is orthogonal to MTEB, indicating traditional passage retrieval performance does not generalize to long-horizon memory. This benchmark fills a critical gap, providing a standardized tool for advancing memory embedding development.

Key takeaway

For AI Engineers developing or selecting embedding models for memory-augmented systems, you should recognize that traditional passage retrieval benchmarks like MTEB are insufficient for long-horizon memory tasks. Utilize the LMEB framework to rigorously evaluate your models across episodic, dialogue, semantic, and procedural memory types. This will help you identify models truly capable of handling complex, context-dependent memory retrieval, as larger models do not guarantee superior performance in these specialized scenarios.

Key insights

LMEB provides a comprehensive benchmark for evaluating embedding models on complex, long-horizon memory retrieval tasks.

Principles

• Zero-shot evaluation is crucial for assessing embedding capabilities.
• Model architecture and task adaptability are key, not just size.
• Long-horizon memory retrieval differs from traditional passage retrieval.

Method

LMEB evaluates 15 models across 22 datasets and 193 zero-shot tasks, categorized into episodic, dialogue, semantic, and procedural memory types, using both AI-generated and human-annotated data.

In practice

• Use LMEB to benchmark models for memory-augmented systems.
• Consider LMEB's four memory types for diverse model testing.
• Don't assume larger models always perform better for memory tasks.

Topics

• Memory Embeddings
• Long-horizon Memory Retrieval
• LMEB Benchmark
• Text Embedding Evaluation
• Agentic Systems
• Zero-shot Learning

Code references

• KaLM-Embedding/LMEB
• KaLM-Embedding/LMEB.github.io

Best for: Research Scientist, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• MemoryDocDataSet: A Benchmark for Joint Conversational Memory and Long Document Reasoning — MemoryDocDataSet uniquely benchmarks AI systems on joint conversational memory and long document reasoning, revealing a critical performance gap.
• VikingMem: A Memory Base Management System for Stateful LLM-based Applications — VikingMem introduces a Memory Base paradigm for stateful LLM interactions, improving memory retrieval and generalizability.
• Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings — Longitudinal evaluation with ground-truth-first generation reveals critical memory architecture performance differences and security properties for LLM agents.
• CoreMem: Riemannian Retrieval and Fisher-Guided Distillation for Long-Term Memory in Dialogue Agents — CoreMem uses information geometry for Riemannian retrieval and Fisher-guided distillation to enable long-term memory on 8 GB VRAM edge devices.
• MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models — MEMLENS benchmarks multimodal long-term memory, revealing distinct strengths and weaknesses in LVLMs and memory-augmented agents.
• AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents — A bounded memory contract for LLM agents, using typed retrieval for fresh context, enables isolated memory component study.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.