GPT-Realtime-2, Directionally Bad and Agent Memory

2026-05-07 · Source: Matthew Berman · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Advanced, extended

Summary

The stream features Richmond De from Oracle, an expert in agent memory, discussing recent advancements and challenges in AI. The conversation highlights Anthropic's "Dreaming" feature in Claude managed agents, which consolidates past session memories to improve self-improvement and response quality, particularly during off-peak hours. OpenAI has also introduced memory improvements, allowing users to upvote or downvote memory sources for personalized experiences. De emphasizes that these developments, while seemingly novel, build upon earlier research like the Stanford Simuaka paper and the MEGPT team's "sleep time compute." The discussion also covers the concept of "memory engineering" as a critical discipline for optimizing AI agent performance, reducing operational costs, and enhancing accuracy, demonstrating how engineered memory can stabilize token consumption and improve LLM-as-a-judge preferred responses over naive approaches.

Key takeaway

For AI Engineers and Machine Learning Engineers building robust AI systems, focusing on memory engineering is paramount. You should explore techniques like context window utility, information compaction, and strategic retrieval to optimize agent performance and manage operational costs. Adopting a single, comprehensive database for heterogeneous data sources can significantly reduce cognitive load and improve retrieval efficiency for your agents.

Key insights

Agent memory and context engineering are crucial for enhancing AI agent performance and reducing operational costs.

Principles

• Memory consolidation improves agent self-improvement.
• Human feedback refines AI memory personalization.
• Memory engineering stabilizes token consumption.

Method

Anthropic's "Dreaming" feature consolidates past session memories during off-peak hours to improve agent self-improvement and response quality, effectively offloading compute demand.

In practice

• Implement memory engineering to stabilize token consumption.
• Utilize human feedback for memory personalization.
• Store tools in a database for scalable access.

Topics

• Agent Memory
• Context Engineering
• Anthropic Dreaming Feature
• OpenAI Memory Improvements
• Continual Learning

Best for: AI Engineer, Machine Learning Engineer, AI Scientist

Related on AIssential

• Agent memory resolved? — A git-like memory framework significantly enhances coding agent performance and context persistence across sessions.
• Engineering Memory for AI Agents: A Practical Beginner’s Guide — AI agent memory selectively retains past interaction details to enable contextual, multi-step task execution.
• Context Engineering for AI Agents: Part 2 — Effective AI agent performance hinges on precise context management, minimizing irrelevant information and optimizing tool access.
• How AI Agents Manage Memory and Avoid Forgetfulness — The perceived memory of AI agents is an engineered system around stateless LLMs, not an inherent model capability.
• Your AI Agent Has a Memory Problem — AI agents require robust memory systems to retain context and improve performance across sessions.
• The man behind Cursor's "memory" feature — Cursor's Memory feature enhances agent context by distinguishing between task-specific and generalizable information for improved coding assistance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Matthew Berman.