Remember to be Curious: Episodic Context and Persistent Worlds for 3D Exploration

2026-05-21 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

A new approach addresses challenges in curiosity-driven reinforcement learning for 3D environments, where agents often get stuck in local loops or revisit forgotten states. This method, detailed in "Remember to be Curious," demonstrates that effective exploration requires both spatial persistence and episodic context. It achieves this by employing an online 3D reconstruction to create a continuously updated, persistent world model, while the agent's policy is parameterized as a sequence model over RGB observations to maintain episodic trajectory history. Trained solely on curiosity using HM3D, this agent significantly outperforms RL-based active mapping baselines and exhibits zero-shot generalization to Gibson and AI-generated worlds. The end-to-end policy also facilitates efficient adaptation to downstream tasks, including apple picking and image-goal navigation.

Key takeaway

For Machine Learning Engineers designing exploration strategies in complex 3D environments, traditional curiosity-driven reinforcement learning often fails due to a lack of spatial persistence and episodic memory. You should consider integrating persistent world models, such as online 3D reconstruction, and episodic context via sequence models into your agent architectures. This approach improves exploration efficiency, enables zero-shot generalization, and accelerates adaptation to diverse downstream tasks like navigation or object interaction.

Key insights

Effective curiosity in 3D environments requires spatial persistence and episodic context to overcome local loops and forgotten states.

Principles

• Curiosity needs a persistent, continuously updated world model.
• Agents must maintain episodic history for novel region navigation.
• Online 3D reconstruction provides spatial persistence.

Method

Implement an online 3D reconstruction for a persistent world model and parameterize the agent policy as a sequence model over RGB observations to maintain episodic context.

In practice

• Train agents purely via curiosity on HM3D.
• Deploy agents using solely RGB frames.
• Adapt policies to downstream tasks efficiently.

Topics

• Reinforcement Learning
• 3D Exploration
• Curiosity-Driven Learning
• Persistent World Models
• Episodic Memory
• Online 3D Reconstruction
• RGB-based Navigation

Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• Can In-Context Learning Support Intrinsic Curiosity? — ICL can support intrinsic curiosity for data selection in non-temporal settings, but not general MDPs.
• Current World Models Lack a Persistent State Core — Current world models lack persistent state evolution when unobserved, a critical gap requiring new design objectives beyond surface fidelity.
• Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning — Preserving diverse policy neighborhoods in a stable latent space enhances continual reinforcement learning plasticity.
• Look Before You Leap: Autonomous Exploration for LLM Agents — Autonomous exploration is crucial for LLM agents to adapt and generalize in unfamiliar environments.
• Mem-World: Memory-Augmented Action-Conditioned World Models for Persistent Robot Manipulation — Mem-World uses a 4D surfel-indexed memory for geometry-aware history retrieval to enable persistent robot manipulation.
• ARROW: Augmented Replay for RObust World models — Augmented replay to a World Model significantly reduces catastrophic forgetting in continual reinforcement learning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.