Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models

2026-07-09 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, medium

Summary

Hidden Decoding is a novel sequence-length scaling method designed to enhance Large Language Models (LLMs) during continued pretraining without requiring costly full pretraining or modifying the Transformer backbone's width. This approach addresses the scalability challenges of depth-recurrent Transformers by expanding each token into n independent streams, each with its own embedding table, and maintaining the intermediate streams' key-value cache as context. This allows for increased internal computation per token along the sequence-length dimension, compatible with standard large-model training. To ensure affordability at scale, Hidden Decoding incorporates Stream-Factorized Attention, which significantly reduces attention costs by primarily attending within streams and only selectively mixing across them. Experiments with WeLM-HD4-80B and WeLM-HD4-617B at n=4 demonstrated improvements over non-HD baselines, establishing Hidden Decoding as the first sequence-length scaling method successfully applied at the 100B+ Mixture-of-Experts (MoE) scale, with gains increasing as n grows.

Key takeaway

For AI Architects evaluating strategies to enhance existing large language models, Hidden Decoding offers a compelling alternative to costly full pretraining or widening models. You should investigate implementing this sequence-length scaling method during continued pretraining. Utilize its token stream expansion and Stream-Factorized Attention to boost performance on 100B+ MoE models. This approach allows you to achieve significant gains by allocating more computation per token without altering the core Transformer backbone.

Key insights

Scaling LLMs by expanding computation along the sequence-length dimension with token streams offers a fixed-backbone improvement path.

Principles

• Sequence-length expansion enables fixed-backbone LLM scaling.
• Pipeline parallelism hinders depth-recurrent Transformer scaling.
• Stream-factorized attention reduces quadratic attention costs.

Method

Hidden Decoding expands each token into n streams using independent embedding tables and stream-factorized attention, keeping intermediate key-value caches as context.

In practice

• Implement Hidden Decoding during continued pretraining.
• Utilize Stream-Factorized Attention for large-scale efficiency.
• Apply to 100B+ MoE models for performance gains.

Topics

• Hidden Decoding
• Large Language Models
• Sequence-Length Scaling
• Continued Pretraining
• Stream-Factorized Attention
• Mixture-of-Experts

Code references

• tjunlp-lab/Lexical-Unit-Decoding-LUD-

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models — Hidden Decoding scales LLMs by increasing per-token computation via sequence-length expansion, improving performance without enlarging the Transformer backbone.
• Challenging Quadratic Attention - A Holistic View On the Rise of Alternative Language Model Architectures — Alternative language model architectures are emerging to challenge Transformer dominance by addressing quadratic attention complexity.
• Compressing Sequences in the Latent Embedding Space: $K$-Token Merging for Large Language Models — K-Token Merging compresses LLM inputs in the latent embedding space for efficiency.
• Large Language Models Make Sense When You See Why Previous AI Couldn’t Handle Language — Transformers' attention mechanism enabled LLMs to overcome sequential processing limitations, allowing for scalable language understanding and generation.
• Scaling Capability in Token Space: An Analysis of Large Vision Language Model — LVLMs exhibit predictable sublinear and linear scaling regimes based on vision token count.
• Algorithmic Progress In Language Models — Algorithmic progress in language models halves compute needs every 8 months but contributes less than compute scaling to overall performance gains.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.