Agent Memory From Scratch

2025-02-06 · Source: Decoding AI Magazine · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Software Development & Engineering · Depth: Advanced, long

Summary

This article details a comprehensive approach to building a unified agent memory from scratch, emphasizing knowledge graphs (KGs) and MongoDB. It outlines an architecture comprising an ontology, a durable write path, a single data collection, and a serving layer. The system leverages MongoDB for integrated text, vector, and graph search, ensuring data lineage. The write path processes documents into 512-token chunks with 64 overlap, extracts KG entities using an LLM ("gemini-3.1-flash-lite"), validates via Pydantic, resolves names, embeds with Voyage "voyage-3.5" (1024 dimensions), and deduplicates based on cosine similarity thresholds (≥0.95 merge, ≤0.85 new node, 0.85-0.95 "same_as" edge). The ontology, based on the POLE+O model, defines "person", "object", "location", "event", "organization", "preference", and "fact" nodes. Retrieval methods include graph search with Reciprocal Rank Fusion (RRF) and 2-hop "\$graphLookup", deep search for LLM wikis, and agentic NL query generation. The author suggests a Level 2 "build vs. buy" strategy, using an SDK like Graphiti or mem0.

Key takeaway

For AI Engineers designing agent memory, deeply understanding the underlying knowledge graph architecture is crucial before adopting off-the-shelf tools. Your choice of ontology, write path, and storage impacts scalability and data integrity. Consider a Level 2 approach, leveraging an SDK for core memory functions while owning business logic and serving layers to balance customization with development speed. Prioritize robust deduplication to prevent irreversible data errors.

Key insights

Building robust agent memory requires understanding underlying KG architecture, ontology, and data flow.

Principles

• Ontology defines the contract for LLM extraction and query.
• Deduplication is critical; wrong merges are unrecoverable.
• Single database simplifies lineage and scales well for many use cases.

Method

The proposed method involves a 7-stage write path: chunking (512 tokens, 64 overlap), LLM extraction, Pydantic validation, name resolution, embedding, deduplication (cosine similarity thresholds), and upserting into a MongoDB collection.

In practice

• Define ontology using Pydantic and POLE+O model.
• Use RRF for fusing vector and text search ranks.
• Implement deduplication with 0.85-0.95 similarity threshold for human review.

Topics

• Agent Memory
• Knowledge Graphs
• MongoDB
• Ontology Engineering
• LLM Data Extraction
• Data Deduplication
• Reciprocal Rank Fusion

Code references

• getzep/graphiti
• mem0ai/mem0
• topoteretes/cognee

Best for: AI Engineer, Machine Learning Engineer, AI Architect

Related on AIssential

• Governed Shared Memory for Multi-Agent LLM Systems — Governed shared memory for multi-agent LLM systems requires explicit systems-level abstractions to prevent critical failure modes.
• Agent Memory: Characterization and System Implications of Stateful Long-Horizon Workloads — LLM agent memory systems' costs shift between read/write paths based on design choices.
• Building Agentic GraphRAG Systems — GraphRAG is fundamentally an ontology-first data modeling problem, not just a retrieval algorithm, crucial for agent unified memory.
• Knowledge Graph Engineering For Agents: The Multidomain Problem — AI agents require explicit engineering to integrate multi-domain knowledge and accumulate persistent memory, unlike human pattern-matching.
• Agent Memory Patterns in Cognitive Science and AI Systems — Effective AI agents require structured memory systems mirroring human cognition to maintain context and learn.
• Wiki Code Memory: Giving My Coding Agent a Real Memory (and Cutting Token Costs While I’m At It) — Combining episodic and semantic memory layers creates a cost-effective, persistently knowledgeable coding agent.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Decoding AI Magazine.