Per-Token Fixed-Point Convergence in Depth-Recurrent Transformers

2026-07-15 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A study on depth-recurrent transformers, which apply a weight-tied core a variable number of times, reveals that their recurrent state converges to a per-token fixed point. On a 135M-class model trained on FineWeb-Edu, mean successive-output KL divergence drops from 3.9e-1 at the second loop to 8.5e-6 by the sixteenth. Crucially, this convergence is non-uniform; the median token converges by loop six, while approximately 10 percent of tokens continue to update at depth eight, with convergence depth ordered by token type. The research introduces a training-free rule that halts each token upon output stabilization, achieving uniform depth-8 quality at 4.94 average loops, a 38 percent reduction in average depth. This rule outperforms a learned linear router. Validation loss decreases monotonically from 3.80 at one loop to 3.20 at eight, remaining stable to 32 loops. The complete study was conducted on a single RTX 4090 over approximately 100 GPU-hours.

Key takeaway

For Machine Learning Engineers optimizing depth-recurrent transformer inference, this research demonstrates a significant opportunity to reduce computational cost. By implementing a training-free halting rule that stabilizes each token's output, you can achieve depth-8 quality with a 38 percent reduction in average loops, translating to substantial FLOP savings. Consider integrating this adaptive per-token halting mechanism to enhance model efficiency without compromising performance.

Key insights

Depth-recurrent transformers exhibit non-uniform per-token fixed-point convergence, enabling efficient, adaptive inference halting.

Principles

• Recurrent states in depth-recurrent transformers converge per token.
• Convergence depth varies significantly by token type.
• Training-free halting rules can surpass learned predictors in efficiency.

Method

A training-free rule halts each token when its output stabilizes, achieving uniform quality at 4.94 average loops, a 38% depth reduction.

In practice

• Implement training-free halting for depth-recurrent models.
• Analyze token-level convergence to optimize inference schedules.

Topics

• Depth-Recurrent Transformers
• Fixed-Point Convergence
• Inference Optimization
• Halting Rules
• Token-level Analysis
• Model Efficiency

Best for: NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield — Compute-efficient LLM training is achieved by concentrating supervision on semantic tokens and architecturally optimizing models via compression and expert fusion.
• MARS: Margin-Adversarial Risk-controlled Stopping for Parallel LLM Test-time Scaling — MARS reduces LLM test-time costs by adaptively stopping parallel reasoning traces while preserving accuracy.
• Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models — Visual token reduction in VLMs should prioritize recoverable routing over irreversible pruning due to dynamic token importance.
• D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding — D-Cut adaptively prunes speculative decoding drafts across batches, focusing verification on high-confidence tokens to boost LLM inference speed.
• ResonatorLM: Causal Resonant Field Mixing for Efficient Long-Context Language Modelin — ResonatorLM replaces self-attention with physics-derived resonant fields, achieving superior long-context efficiency and accuracy.
• Towards Selective Depth Mixing: Attention Residuals for Stable, Reusable Features in Deep Transformers — Novel architectural and inference techniques enhance LLM efficiency and performance by optimizing residual connections, KV cache management, and sampling.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.