CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention

2026-07-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

CARVE (Content-Aware Recurrent with Value Efficiency) is a novel architecture addressing key limitations in advanced recurrent language models like GDN-2. It resolves three intrinsic defects of prior recurrent sequence models (like GDN-2): memory-blind gating, write-gate bandwidth bottlenecks, and value-axis erase destroying chunk-parallelism. CARVE introduces content-aware erase, reusing recurrent output tensors ("m_c") for memory-conditioned selectivity at zero additional HBM cost, with a measured 0.18% gate perturbation across chunk lengths up to L=128. It also implements a scalar value write gate, reducing parameters from 589,824 to 9,216 per layer without associative capacity loss. At the 1.3B-parameter scale, trained on 100B tokens, CARVE achieves a WikiText perplexity of 15.72 (vs. GDN-2's 15.90), leads common-sense reasoning by +0.63 pp, and sets new benchmarks on RULER retrieval. Critically, it maintains throughput within 0.4% of the baseline while reducing peak memory by 13% and mixer parameters by 19%.

Key takeaway

For Machine Learning Engineers designing or optimizing recurrent neural networks, CARVE presents a compelling architecture for long-context language models. Its content-aware erase and scalar write gate deliver superior perplexity and retrieval performance while significantly reducing memory footprint by 13% and mixer parameters by 19%. You should evaluate CARVE's key-axis gating principle to achieve efficient chunk-parallel training and enhanced memory management in your next-generation recurrent models, especially for tasks demanding precise information retention.

Key insights

Recurrent models can achieve content-aware forgetting and efficiency by reusing kernel outputs and restricting gating to the key axis.

Principles

• Key-axis gating enables chunk-parallel training.
• Output reuse provides memory-awareness at zero HBM cost.
• Scalar write gates suffice for associative recall.

Method

CARVE restricts gating to the key axis, uses a chunk-mean of previous recurrent outputs ("m_c") for content-aware erase, and employs a scalar write gate per head, preserving WY-form chunk-parallel training.

In practice

• Implement key-axis-only gating for recurrent models.
• Reuse kernel outputs for memory-aware gate conditioning.
• Consider scalar write gates for parameter reduction.

Topics

• Recurrent Neural Networks
• Linear Attention
• Memory Management
• Chunk-Parallel Training
• Hardware Efficiency
• Language Modeling

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention — CARVE improves recurrent models by content-aware key-axis memory erasure, enabling efficient chunk-parallel training and superior performance.
• TFGN: Task-Free, Replay-Free Continual Pre-Training Without Catastrophic Forgetting at LLM Scale — TFGN enables replay-free, task-free continual pre-training for LLMs by architecturally separating read and write operations.
• Google Published a Paper That Might End the Transformer-Only LLM Era — Recurrent models can achieve growing memory by caching compressed states from sequence segments, bridging RNN and Transformer memory paradigms.
• Forget, Then Recall: Learnable Compression and Selective Unfolding via Gist Sparse Attention — GSA uses learnable gist tokens to compress context and selectively unfold relevant raw chunks, achieving efficient long-context LLM processing.
• STaR-KV: Spatio-Temporal Adaptive Re-weighting for KV Cache Compression in GUI Vision-Language Models — GUI VLM KV cache compression benefits from spatio-temporal adaptive re-weighting, moving beyond fixed saliency and cutoffs.
• Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generation — Sparse Forcing improves long-horizon video generation quality and efficiency by leveraging trainable sparse attention and persistent memory.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.