Mem0 vs Zep vs Letta: A Folder of Text Files Shouldn't Beat the 61K-Star Memory Layer

2026-07-27 · Source: Towards AI - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Advanced, quick

Summary

A recent analysis of agent memory layers, including Mem0, Zep, and Letta, reveals significant issues within the field's benchmarking and system reliability. Despite Mem0's 61K GitHub stars, a simple file system with "grep" (Letta) surprisingly surpassed Mem0's published LoCoMo score, achieving 74.0% against Mem0's 68.5%. Furthermore, a 20-probe test demonstrated that semantic retrieval systems failed to recall updated facts in 19 out of 20 instances. The investigation highlights a critical lack of independent reproduction for agent memory performance numbers, with a major benchmark found to contain an arithmetic error that inflated a headline score by 25.56 percentage points.

Key takeaway

For AI Engineers evaluating agent memory layers, you should approach published benchmark scores with extreme skepticism, as independent validation is critically absent and errors are prevalent. Prioritize implementing your own 20-probe tests to verify a system's ability to handle fact changes, rather than relying on vendor claims. You might find that simpler, file-based memory solutions offer more reliable performance for specific tasks than complex, highly-starred alternatives.

Key insights

Agent memory benchmarks lack independent validation, and current semantic retrieval systems struggle with fact updates.

Principles

• Independent reproduction is critical for benchmark validity.
• Simple methods can outperform complex memory layers.
• Semantic retrieval often fails on fact updates.

Method

A 20-probe test can evaluate memory systems' ability to recall changed facts, using a simple harness to assess stack performance.

In practice

• Test memory systems for fact update recall.
• Consider simple file-based memory for agents.
• Verify benchmark claims independently.

Topics

• Agent Memory
• LLM Agents
• Benchmarking
• Semantic Retrieval
• Mem0
• Zep
• Letta

Best for: AI Architect, Research Scientist, CTO, AI Engineer, Machine Learning Engineer, AI Scientist

Related on AIssential

• Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings — Longitudinal evaluation with ground-truth-first generation reveals critical memory architecture performance differences and security properties for LLM agents.
• Honcho vs Mem0: Two Memory Layers, Two Architectures — Honcho and Mem0 differ in memory architecture: Honcho stores peer-tagged conclusions, Mem0 stores user-tagged facts.
• HippoCamp: Benchmarking Contextual Agents on Personal Computers — Current AI agents struggle with multimodal perception and evidence grounding in realistic personal file management.
• Agent Memory Systems and Knowledge Graphs: Letta, Mem0, Graphiti, and Cognee — Agent memory systems balance graph complexity, cost, and retrieval capabilities for persistent knowledge.
• MemTrace: Probing What Final Accuracy Misses in Long-Term Memory — MemTrace evaluates LLM long-term memory by probing knowledge points across controlled dimensions, revealing hidden failure modes beyond aggregated accuracy.
• Graph-Validated Memory Architecture: Enhancing Contextual Accuracy and Retrieval Speed in Agentic… — The Graph-Validated Memory Architecture resolves the latency-accuracy trade-off in agentic systems by combining fast semantic caching with knowledge graph validation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards AI - Medium.