CoMeT: Collaborative Memory Transformer for Efficient Long Context Modeling

2026-04-21 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

The Collaborative Memory Transformer (CoMeT) is a novel architecture designed to enable Large Language Models (LLMs) to process arbitrarily long sequences with constant memory usage and linear time complexity, addressing the quadratic complexity and growing key-value (KV) cache issues of standard Transformers. CoMeT operates as an efficient, plug-in module, integrating into pre-trained models with minimal fine-tuning. It employs a dual-memory system: a temporary FIFO queue for recent events and a global memory with a gated update rule for long-range dependencies, which act as a dynamic soft prompt for subsequent data chunks. A layer-level pipeline parallelism strategy facilitates efficient fine-tuning on extremely long contexts. CoMeT, fine-tuned on 32k contexts, can accurately retrieve a passkey from any position within a 1M token sequence, achieving a 21x inference speedup and 10x smaller memory footprint compared to a full-attention baseline. It also surpasses other efficient methods on the Scrolls benchmark and performs comparably to full-attention baselines on summarization tasks, with practical effectiveness validated on real-world agent and user behavior QA tasks.

Key takeaway

For NLP Engineers and Research Scientists working with LLMs on long-context tasks, CoMeT offers a practical solution to overcome the quadratic complexity and memory limitations of standard Transformers. Its dual-memory system and efficient training strategy allow for processing sequences up to 1M tokens with significantly reduced inference time and memory footprint. You should consider integrating CoMeT to enhance the scalability and performance of your LLM applications, especially for summarization, question answering, and agent tasks requiring extensive context.

Key insights

CoMeT enables LLMs to process arbitrarily long sequences with constant memory and linear time via a dual-memory system.

Principles

• Dual-memory systems balance recent detail and long-term retention.
• Gated updates protect salient historical information from overwriting.
• Layer-level pipeline parallelism enhances distributed training efficiency.

Method

CoMeT processes data in chunks, prepending global and temporary memories to hidden states. Global memory uses a Residual Low-Rank Adapter (RLA) and a gating mechanism for updates, while temporary memory is a fixed-capacity FIFO queue of RLA-processed compression tokens.

In practice

• Integrate CoMeT into pre-trained LLMs with minimal fine-tuning.
• Utilize layer-level pipeline parallelism for efficient training on 128k+ token contexts.
• Employ CoMeT for tasks requiring extreme long-context understanding, like 1M token retrieval.

Topics

• Collaborative Memory Transformer
• Long Context Modeling
• Dual-Memory System
• Layer-Level Pipeline Parallelism
• Transformer Efficiency

Code references

• iflow-ai/iflow-cli
• laude-institute/terminal-bench

Best for: NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• End-to-End Context Compression at Scale — Encoder-decoder compression can surpass KV cache methods for efficient, high-quality long-context LLM inference.
• Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization — Parallel Track Transformers reduce GPU synchronization for faster, more efficient LLM inference.
• MemoryDocDataSet: A Benchmark for Joint Conversational Memory and Long Document Reasoning — MemoryDocDataSet uniquely benchmarks AI systems on joint conversational memory and long document reasoning, revealing a critical performance gap.
• Google’s Titans: 4 Ways to Wire Long-Term Memory Into a Transformer — Titans integrates a separate neural memory module into Transformers for long-term, selective information retention.
• Building a Context Pruning Pipeline for Long-Running Agents — Efficiently manage LLM context for long-running agents by dynamically pruning conversation history based on semantic relevance.
• Ulysses Sequence Parallelism: Training with Million-Token Contexts — Ulysses SP enables training LLMs with million-token contexts by distributing attention heads across GPUs, significantly reducing memory and boosting throughput.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.