DeferMem: Query-Time Evidence Distillation via Reinforcement Learning for Long-Term Memory QA

2026-05-22 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

DeferMem is a novel long-term memory framework designed to enhance large language model (LLM) agents' question answering over extensive conversational histories. It addresses the challenge of scattered and irrelevant evidence by decoupling memory utilization into high-recall candidate retrieval and query-conditioned evidence distillation. The system employs a lightweight segment-link structure to organize raw history and retrieve broad candidate sets at query time. Subsequently, a memory distiller, trained with the DistillPO reinforcement learning algorithm, refines these noisy candidates into faithful, self-contained, and query-conditioned evidence. DistillPO utilizes a structured action space, a decomposed-and-gated reward pipeline, and structure-aligned advantage assignment. On LoCoMo and LongMemEval-S benchmarks, DeferMem achieved 70.0% and 88.25% non-adversarial accuracy respectively, demonstrating superior QA performance and memory-system efficiency with the fastest runtime and zero commercial-API token cost for memory operations.

Key takeaway

For AI Scientists and Machine Learning Engineers developing LLM agents with long-term memory, consider adopting a query-time evidence distillation approach. DeferMem demonstrates that decoupling high-recall retrieval from a reinforcement learning-trained distiller significantly boosts QA accuracy and operational efficiency, achieving zero commercial-API token cost. You should explore structured RL methods like DistillPO to refine noisy retrieved content, ensuring your agents can effectively utilize vast conversational histories without performance degradation.

Key insights

Query-time evidence distillation via reinforcement learning significantly improves LLM agent long-term memory QA efficiency and accuracy.

Principles

• Defer evidence distillation until query time.
• Decouple retrieval from evidence refinement.
• Use structured actions for complex RL tasks.

Method

DeferMem uses a segment-link structure for high-recall retrieval, then a DistillPO-trained distiller performs message selection and rewriting, optimized by decomposed-gated rewards and structure-aligned advantage assignment.

In practice

• Implement segment-link structures for high-recall memory retrieval.
• Train LLM distillers with structured RL for noisy candidate refinement.
• Decompose rewards for complex, multi-stage generation tasks.

Topics

• LLM Agents
• Long-Term Memory
• Question Answering
• Reinforcement Learning
• Evidence Distillation
• Retrieval-Augmented Generation

Code references

• huggingface/trl

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• DeferMem: Query-Time Evidence Distillation via Reinforcement Learning for Long-Term Memory QA — DeferMem improves LLM long-term memory QA by decoupling retrieval from query-conditioned evidence distillation using RL.
• MemTrace: Probing What Final Accuracy Misses in Long-Term Memory — MemTrace evaluates LLM long-term memory by probing knowledge points across controlled dimensions, revealing hidden failure modes beyond aggregated accuracy.
• Thinking with Reasoning Skills: Fewer Tokens, More Accuracy — Distilling and retrieving reusable reasoning skills significantly improves LLM efficiency and performance.
• MARDoc: A Memory-Aware Refinement Agent Framework for Multimodal Long Document QA — MARDoc uses a structured, dynamically updated memory to overcome context dilution in multimodal long-document QA agents.
• Phase Transitions in Agent Memory: Recurrent Memory — Long-running LLM agents benefit from density-driven memory consolidation, triggering processing only upon sustained semantic recurrence.
• Recursive Think-Answer Process to Stop "Thinking" Earlier — Integrating confidence-aware self-correction and RL-aware distillation improves LLM reasoning and efficiency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.