Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models

2026-07-09 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

Hidden Decoding is a novel sequence-length scaling method designed to improve Large Language Models (LLMs) during continued pretraining without enlarging the Transformer backbone. This approach expands each token into "n" independent streams, utilizing separate embedding tables and maintaining the intermediate streams' key-value cache as context, thereby increasing internal computation per token. To manage computational costs at scale, Hidden Decoding incorporates Stream-Factorized Attention, which significantly reduces attention complexity from quadratic to roughly linear in "n" by limiting cross-stream attention to only a few layers. Experiments demonstrate its effectiveness at frontier scale, with models like WeLM-HD4-80B and WeLM-HD4-617B showing improvements over their non-HD baselines when trained at n=4. This marks Hidden Decoding as the first sequence-length scaling method proven at the 100B+ Mixture-of-Experts (MoE) scale, confirming that gains increase with higher "n" values.

Key takeaway

For Machine Learning Engineers optimizing large language models, if you are seeking to enhance an existing LLM's performance without incurring the immense cost of full backbone re-pretraining, consider implementing Hidden Decoding. This method allows you to scale computation per token effectively, improving model capabilities. You should explore its application during continued pretraining, utilizing Stream-Factorized Attention to maintain computational efficiency, especially for models at the 100B+ MoE scale.

Key insights

Hidden Decoding scales LLMs by increasing per-token computation via sequence-length expansion, improving performance without enlarging the Transformer backbone.

Principles

• Fixed-backbone scaling improves LLMs without costly full pretraining.
• Sequence-length expansion offers a viable scaling path for frontier LLMs.
• Attention cost can be optimized by factorizing across computation streams.

Method

Hidden Decoding expands each token into 'n' streams with independent embeddings and context. Stream-Factorized Attention limits cross-stream attention to few layers, reducing cost from quadratic to linear in 'n'.

In practice

• Apply Hidden Decoding during continued pretraining for LLM improvement.
• Utilize Stream-Factorized Attention to manage computational costs.
• Explore higher 'n' values for increased performance gains.

Topics

• Hidden Decoding
• Latent Computation Scaling
• Large Language Models
• Stream-Factorized Attention
• Continued Pretraining
• Transformer Architectures

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models — Scaling LLMs by expanding computation along the sequence-length dimension with token streams offers a fixed-backbone improvement path.
• Context compression finally works in production: new research cuts LLM input 16x without the accuracy hit — Latent Context Language Models compress LLM input context before decoding, significantly boosting speed and memory efficiency without major accuracy loss.
• When Data Is Scarce: Scaling Sparse Language Models with Repeated Training — Sparse training improves LLM scaling trade-offs and delays data saturation in data-scarce environments.
• The Evolution of LLM Inference: Decoding algorithms — Part 1 — LLM inference speed hinges on reducing sequential decoding steps by predicting and verifying multiple tokens in parallel.
• Compute Optimal Tokenization — Optimal language model scaling depends on data in bytes, not tokens, with a compute-dependent optimal compression rate.
• SeLaR: Selective Latent Reasoning in Large Language Models — SeLaR improves LLM reasoning by selectively applying soft embeddings and regularization to enhance exploration at low-confidence steps.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.