Memory-R2: Fair Credit Assignment for Long-Horizon Memory-Augmented LLM Agents

2026-05-22 · Source: cs.MA updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

Memory-R2 is a novel training framework designed for long-horizon memory-augmented Large Language Model (LLM) agents, addressing the critical challenge of fair credit assignment in multi-session reinforcement learning. Its core, LoGo-GRPO, integrates global trajectory-level optimization with local rerollouts from shared intermediate memory states, ensuring more precise supervision for memory construction. The framework also features a shared-parameter co-learning design, where a fact extractor and memory manager, instantiated from a single LLM backbone via role-specific prompts, jointly optimize memory formation and evolution. To stabilize training over extended memory horizons, Memory-R2 employs a progressive curriculum, scaling from 8 to 16 to 32 sessions. Benchmarking on LoCoMo, Memory-R2 with a Qwen2.5-7B-Instruct backbone achieved an F1 score of 50.60 and B1 of 44.01, outperforming Memory-R1's 43.14 F1 and 36.44 B1. It also demonstrated strong generalization across various benchmarks, model scales, and answer agents, notably improving Qwen2.5-3B's F1 from 10.3 to 46.8, while reducing inference latency.

Key takeaway

For Machine Learning Engineers developing long-horizon LLM agents, you should adopt training frameworks that explicitly address credit assignment challenges in multi-session environments. Implement LoGo-GRPO's global-local optimization and a progressive curriculum to stabilize training and improve memory quality. This approach enables robust, data-efficient memory construction, even for smaller models, and can reduce inference latency, making your agents more reliable and performant over extended interactions.

Key insights

Fair credit assignment and progressive training are crucial for long-horizon memory-augmented LLM agents.

Principles

• Memory makes RL environments non-stationary.
• Local rerollouts improve credit assignment fairness.
• Curriculum learning stabilizes long-horizon RL.

Method

Memory-R2 uses LoGo-GRPO, combining global trajectory rewards with local rerollouts from shared memory states. It employs a shared LLM backbone for fact extraction and memory management, and a progressive curriculum (8->16->32 sessions) for stable long-horizon RL.

In practice

• Implement local rerollouts for fairer RL credit.
• Use a progressive curriculum for long-horizon training.
• Co-learn memory roles with shared LLM parameters.

Topics

• Memory-Augmented LLMs
• Reinforcement Learning
• Credit Assignment
• Curriculum Learning
• LoGo-GRPO
• Memory Management Policies

Code references

• ahmedehabb/Memory-R2

Best for: Research Scientist, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards — SC-GRPO improves LLM reasoning by using self-conditioned KL divergence for fine-grained credit assignment in RLVR.
• Hindsight Credit Assignment for Long-Horizon LLM Agents — HCAPO uses LLMs as hindsight critics to refine step-level Q-values, improving credit assignment in long-horizon tasks.
• Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning — A diverse data recipe alone significantly improves long-context reasoning in reinforcement learning for LLMs.
• HiMPO: Hindsight-Informed Memory Policy Optimization for Less-Entangled Credit in Long-Horizon Agents — HiMPO disentangles memory credit assignment in long-horizon agents using local utility and hindsight relevance for improved performance.
• Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization — CW-GRPO improves LLM search agents by using an LLM judge for fine-grained, per-round credit assignment.
• Foundations of Reinforcement Learning — Reinforcement learning is a distinct ML paradigm where agents learn optimal actions through evaluative feedback and interaction.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.MA updates on arXiv.org.