Error Highways: Scaling Predictive Coding to Very Deep Networks

2026-06-22 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, medium

Summary

Error Highways: Scaling Predictive Coding to Very Deep Networks introduces Highway Error Propagation (HEP), a novel scheme designed to overcome the depth limitations of predictive coding networks (PCNs). PCNs offer a biologically-plausible, local-learning alternative to back-propagation but have been largely restricted to shallow architectures due to rapid decay of the learning signal in deeper layers. HEP addresses this by augmenting the PC free energy function with feedback matrices V_{L\to i}, which directly couple selected hidden states to the clamped output error. This linear coupling ensures that correction signals maintain a magnitude independent of network depth, effectively bypassing the Jacobian chain while preserving the local PC synaptic update rule. The method successfully trains Multi-Layer Perceptrons (MLPs) of up to 128 layers on MNIST and Fashion-MNIST datasets, demonstrating robust accuracy across varying depths.

Key takeaway

For Machine Learning Engineers exploring biologically-plausible learning, Highway Error Propagation (HEP) offers a viable path to scale predictive coding networks beyond shallow architectures. You can now train deep MLPs, up to 128 layers, with PCNs on tasks like image classification, achieving robust accuracy. This method provides a local-learning alternative to back-propagation, potentially simplifying model development and reducing computational overhead for certain deep learning applications.

Key insights

Highway Error Propagation enables deep predictive coding networks by ensuring consistent error signal propagation regardless of depth.

Principles

• Error signal decay limits deep PCNs.
• Direct feedback paths stabilize learning.
• Linear coupling maintains error magnitude.

Method

Augment PC's free energy with feedback matrices V_{L\to i} to linearly couple hidden states to output error, bypassing the Jacobian chain for depth-independent correction.

In practice

• Train MLPs up to 128 layers with PCNs.
• Apply to MNIST and Fashion-MNIST tasks.

Topics

• Predictive Coding Networks
• Deep Neural Networks
• Error Propagation
• Biologically Plausible AI
• MNIST
• Fashion-MNIST

Code references

• imankgoyal/NonDeepNetworks

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes — TBP-mHC provides an exact, fully expressive, and minimal parameterization for stable hyper-connections in deep neural networks.
• Mechanistic estimation for wide random MLPs — Mechanistic estimation for wide random MLPs significantly outperforms Monte Carlo sampling in efficiency and accuracy.
• Semi-Supervised Learning on Graphs using Graph Neural Networks — GNN performance in semi-supervised learning is bounded by approximation, stochastic, and optimization errors, influenced by label scarcity and graph topology.
• Closed-form predictive coding via hierarchical Gaussian filters — Closed-form predictive coding via hierarchical Gaussian filters enables deep, efficient, and biologically plausible neural network training.
• A Practitioner's Guide to Kolmogorov-Arnold Networks — KANs offer enhanced expressivity and interpretability over MLPs by using learnable univariate basis functions on network edges.
• Hybridizing Equilibrium Propagation with Ising Machines for Efficient Energy-Based Learning — Hybridizing EP with Ising dynamics improves energy-based learning by overcoming local minima and accelerating convergence.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.