MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents

2026-05-18 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

MementoGUI is a novel plug-in agentic memory framework designed to enhance Multimodal Large Language Model (MLLM)-based Graphical User Interface (GUI) agents in long-horizon tasks. It addresses the limitations of existing agents that struggle with maintaining task state across numerous interface transitions, often relying on raw history replay or text-only memory. MementoGUI introduces MementoCore, a learned controller that manages online memory selection, compression, and retrieval. This system selectively preserves task-relevant interface events, including textual summaries and Region-of-Interest (ROI)-level visual evidence, in working memory, while episodic memory retrieves reusable past trajectories. The framework also includes a scalable data curation pipeline for training memory controllers and MementoGUI-Bench, a new benchmark for evaluating long-horizon GUI agent decision-making. Experiments across GUI-Odyssey, MM-Mind2Web, and MementoGUI-Bench demonstrate MementoGUI's superior performance compared to baselines.

Key takeaway

For research scientists developing GUI agents for complex, multi-step tasks, MementoGUI offers a robust solution to overcome memory limitations. You should consider integrating agentic memory control to manage interaction history, preserving critical visual and textual context without overwhelming the model. This approach can significantly improve task completion rates and agent reliability in long-horizon scenarios.

Key insights

MementoGUI improves long-horizon GUI agents by learning to selectively manage multimodal memory, preventing information overload and loss.

Principles

• Treat long-horizon GUI control as an online memory-control problem.
• Modularize memory control into specialized operators.

Method

MementoGUI uses MementoCore for online memory selection, compression, and retrieval, preserving task-relevant visual and textual evidence in working memory and retrieving past trajectories from episodic memory.

In practice

• Augment MLLM-based GUI agents with plug-in memory control.
• Utilize ROI-level visual evidence for future decisions.

Topics

• MementoGUI Framework
• Agentic Memory Control
• Long-Horizon GUI Agents
• Multimodal Memory
• MementoCore Controller

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields — Current AI agents struggle with long-horizon, domain-specific GUI workflows, achieving only ~30% success.
• MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models — MEMLENS benchmarks multimodal long-term memory, revealing distinct strengths and weaknesses in LVLMs and memory-augmented agents.
• Agent memory resolved? — A git-like memory framework significantly enhances coding agent performance and context persistence across sessions.
• New framework lets AI agents rewrite their own skills without retraining the underlying model — Memento-Skills enables AI agents to autonomously evolve their capabilities by rewriting their own executable skills.
• OCR-Memory: Optical Context Retrieval for Long-Horizon Agent Memory — Visual encoding of agent trajectories enables long-horizon memory with reduced token costs and hallucination.
• MemGraphRAG: Memory-based Multi-Agent System for Graph Retrieval-Augmented Generation — MemGraphRAG uses a memory-based multi-agent system to build high-quality, globally consistent knowledge graphs for enhanced RAG.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.