PReM: Learning What to Preserve and When to Refresh for Context Compression

2026-07-15 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

PReM (Preserve and Refresh Memory) is a novel context-compression framework designed to enhance long-context inference by dynamically managing the model's internal layer-wise key-value (KV) memory. Unlike existing methods that make early or external compression decisions, PReM learns what contextual information to preserve and when to refresh it. It employs a dedicated memory layer for selection and a special memory token to trigger refreshes during generation. The framework is trained using Phase-Separated Refresh Training, which aligns memory selection with memory-conditioned generation while ensuring continuity. Experiments demonstrate PReM's superior performance over strong baselines, achieving better answer quality and inference efficiency under both 16x and 32x compression with 32K-token contexts.

Key takeaway

For Machine Learning Engineers optimizing long-context inference, PReM offers a dynamic memory management approach that surpasses static compression methods. You should investigate PReM's architecture and Phase-Separated Refresh Training to improve both the efficiency and answer quality of your large language models, especially when dealing with 32K-token contexts and high compression ratios like 16x or 32x.

Key insights

PReM dynamically learns to preserve and refresh context in KV memory for efficient long-context inference.

Principles

• Existing compression methods struggle with adaptive context needs.
• Dynamic memory management improves long-context inference.

Method

PReM uses a memory layer for selection and a memory token for refresh, trained via Phase-Separated Refresh Training to align selection with generation.

In practice

• Achieves 16x and 32x compression for 32K-token contexts.
• Outperforms baselines in answer quality and inference efficiency.

Topics

• Context Compression
• KV Cache
• Long-Context Inference
• PReM
• Memory Management
• Phase-Separated Refresh Training

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Memory-Managed Long-Context Attention: Bounded Editable Memory with a Hard Lifecycle and Calibrated Sparse Fallback — Long-context models need explicit memory management for writes, overwrites, and eviction, separate from state compression.
• PolyKV: Heterogeneous Retention and Allocation for KV Cache Compression — PolyKV optimizes KV cache compression by applying heterogeneous, layer-wise policies and non-uniform budget allocation, significantly improving long-context LLM performance.
• Meta-Soft: Leveraging Composable Meta-Tokens for Context-Preserving KV Cache Compression — Dynamic, composable meta-tokens can compress LLM KV caches while preserving context, outperforming static eviction.
• Reinforced Fast Weights with Next-Sequence Prediction — Next-sequence prediction via reinforcement learning improves fast weight models' long-context understanding.
• Learning When to Attend: Conditional Memory Access for Long-Context LLMs — Conditional attention mechanisms can significantly extend LLM context windows while improving efficiency.
• Context Memorization for Efficient Long Context Generation — Attention-state memory externalizes LLM prefixes into a lookup-based memory, improving long-context inference efficiency and accuracy.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.