PARTREP: Learning What to Repeat for Decoder-only LLMs

2026-07-02 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

PARTREP is a novel selective augmentation method designed to address the asymmetric information flow in decoder-only Large Language Models (LLMs), where later tokens benefit from richer contextual grounding. While full prompt repetition improves reasoning by redistributing context, it significantly increases KV cache footprint and attention costs. PartRep mitigates this by appending only the most informative tokens, identified using token-wise negative log-likelihood (NLL) as a selection signal. To optimize performance, a lightweight gate is trained to predict high-NLL tokens from early-layer hidden states, enabling efficient token selection during mid-prefill via early exit. Evaluated across eight benchmarks, including MMLU, GSM8K, and RULER, and three model families (Qwen2.5, Llama3.2, Gemma4), PartRep achieves most of the performance gains of full repetition while reducing KV cache usage to 59.4% and prefill FLOPs to 79.0%. The paper is set to be published on 2026-07-02.

Key takeaway

For Machine Learning Engineers optimizing decoder-only LLMs for long-context applications, PartRep presents a compelling alternative to full prompt repetition. You can achieve substantial reductions in KV cache footprint (59.4%) and prefill FLOPs (79.0%) while retaining most reasoning performance gains. Consider integrating selective token repetition methods like PartRep to enhance efficiency and scalability in your LLM deployments, especially where computational resources are a constraint.

Key insights

PartRep selectively repeats informative tokens to improve LLM reasoning efficiency by mitigating asymmetric information flow.

Principles

• Asymmetric information flow impacts decoder-only LLMs.
• Less predictable tokens benefit most from late repetition.
• Early-layer hidden states can predict token informativeness.

Method

PartRep uses token-wise NLL to identify informative tokens. A lightweight gate, trained on early-layer hidden states, predicts high-NLL tokens, enabling selection and early exit during mid-prefill to reduce computational cost.

In practice

• Reduce KV cache footprint for LLM inference.
• Improve reasoning in long-context LLMs.
• Optimize prefill FLOPs for decoder-only models.

Topics

• PARTREP
• Decoder-only LLMs
• KV Cache Optimization
• Prompt Engineering
• LLM Inference
• Negative Log-Likelihood

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Lossless Prompt Compression via Dictionary-Encoding and In-Context Learning: Enabling Cost-Effective LLM Analysis of Repetitive Data — LLMs can learn dictionary encodings in-context, enabling lossless prompt compression without fine-tuning and preserving analytical accuracy.
• LLM Inference and Optimization: Fundamentals, Bottlenecks, and Techniques — Optimizing LLM inference requires understanding distinct prefill and decode phases, leveraging KV caching, and applying advanced techniques.
• KVEraser: Learning to Steer KV Cache for Efficient Localized Context Erasing — KVEraser efficiently erases LLM context by replacing KV states with learned steering states, avoiding costly full recomputation.
• Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation — Moment-KV improves LLM long-generation fidelity by using momentum-driven attention aggregation for decoding-time KV cache compression.
• Prompt Repetition: Google's Surprisingly Strong Baseline for Non-Reasoning LLMs — Repeating prompts can dramatically improve LLM accuracy by providing a "clean second read" with full context.
• Breaking the Autoregressive Chain: Hyper-Parallel Decoding for Efficient LLM-Based Attribute Value Extraction — Hyper-Parallel Decoding accelerates LLM inference by exploiting output independence for parallel token generation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.