TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes

2026-05-23 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

TBP-mHC introduces Transportation Birkhoff Polytope (TBP) parameterizations and their Recursive variants (RTBP) to construct exactly doubly stochastic mixing matrices for Hyper-Connections (HC) in residual networks. This approach addresses limitations of prior methods like mHC (approximate Sinkhorn normalization), mHC-lite (factorial complexity), and KromHC (restricted expressivity). TBP-mHC achieves full expressivity of the Birkhoff polytope with minimal (n-1)² degrees of freedom, avoiding iterative normalization and combinatorial explosion. RTBP further enhances speed through hierarchical decomposition and partial parallelization. Empirical evaluations on language model pre-training, using Small (6 Transformer layers, 8 attention heads, 512 embedding) and Medium (12 layers, 12 heads, 768 embedding) configurations with 4 residual streams, demonstrate competitive performance, improved training stability with lower gradient norms, and scalability. The code is publicly available.

Key takeaway

For AI Scientists and Machine Learning Engineers designing deep neural networks with Hyper-Connections, you should consider implementing TBP-mHC or RTBP. These methods provide exact doubly stochastic mixing matrices with full expressivity and minimal parameters, offering superior training stability and competitive performance over approximate or restricted alternatives like mHC and KromHC. Adopting (R)TBP, especially oRTBP variants, can lead to more robust language model pre-training, though be mindful of the non-trivial couplings and nonlinear structure for very large 'n'.

Key insights

TBP-mHC provides an exact, fully expressive, and minimal parameterization for stable hyper-connections in deep neural networks.

Principles

• Doubly stochastic matrices ensure stability in deep network compositions.
• Full expressivity of the Birkhoff polytope is vital for optimal mixing.
• Exact constraint satisfaction significantly improves training stability.

Method

TBP constructs doubly stochastic matrices by generating elements row-by-row from free parameters, updating remaining row/column budgets. RTBP recursively decomposes the matrix into blocks for parallelization.

In practice

• Implement TBP or RTBP to achieve stable, expressive residual mixing.
• Utilize oRTBP with separate optimizer groups for improved training dynamics.
• Apply scaled or margined TBP variants to mitigate gradient issues.

Topics

• Hyper-Connections
• Residual Networks
• Birkhoff Polytope
• Transportation Polytope
• Language Models
• Gradient Stability

Code references

• alyubinin/tbp-mHC
• karpathy/nanoGPT

Best for: NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• The y=x Problem: Rewiring Transformers with Hyper Connections — Hyper Connections and mHC offer new connectivity schemes to stabilize and optimize deep neural network training.
• Expressivity of congruence-based architectures for DNNs on positive-definite matrices — Semi-orthogonality in congruence layers limits DNN expressivity for SPD matrix classification due to spectral diversity loss.
• ResNets, Hyper-Connections, and Manifold Constraints: A Story about Stability — Architectural stability techniques like residual and hyper-connections are crucial for scaling deep neural networks.
• DeepSeek introduces Manifold-Constrained Hyper-Connections for R2 — Manifold-Constrained Hyper-Connections (mHC) enable scalable, cost-effective large language model training by optimizing signal preservation.
• Improved state mixing in higher-order and block diagonal linear recurrent networks — Richer state mixing in LRNNs enhances expressivity and efficiency for long-sequence modeling.
• Equilibrium Propagation and Hamiltonian Inference in the Diffusive Fitzhugh-Nagumo Model — Equilibrium Propagation and Hamiltonian Echo Backpropagation can be extended to skew-gradient systems and deep Energy-Based Models for efficient credit assignment and inference.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.