The Equilibrium Is the Initialization: Lazy Identity Collapse in Physics-Structured Deep Equilibrium Reasoning

2026-07-13 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, medium

Summary

A cautionary study investigates a port-Hamiltonian Deep Equilibrium Model (DEQ) with a learned initialization, revealing that its implicit computation often acts as a "silent no-op." Across two reasoning tasks, ProofWriter entailment and BFS-verified graph-reachability, the solved equilibrium consistently matched the solver's start point to numerical precision. Bypassing the solver entirely resulted in a +0.00 percentage point change in test accuracy in 18 of 19 training runs. Controlled experiments, including removing the anchoring term or retraining with noise-decoupled starts, reproduced these findings. The DEQ's iteration counts showed no correlation with ground-truth difficulty ($r=0.009$), and the full model never outperformed a two-layer MLP. The mechanism is traced to gradient starvation, and the study introduces a four-test diagnostic protocol for auditing claimed implicit computation.

Key takeaway

For AI Scientists and Machine Learning Engineers evaluating or implementing Deep Equilibrium Models, this study reveals that implicit computation can silently fail due to "lazy identity collapse" and gradient starvation. You should apply the proposed four-test diagnostic protocol to rigorously audit DEQ behavior, especially when assessing whether your model genuinely performs iterative inference. Do not solely rely on standard zeroing ablations, as they can be misleading and seed-dependent.

Key insights

Physics-structured Deep Equilibrium Models can exhibit "lazy identity collapse," where implicit computation is a silent no-op due to gradient starvation.

Principles

• DEQs can converge to initialization, bypassing implicit computation.
• Gradient starvation can cause implicit computation to fail.
• Standard zeroing ablations can be confounded and seed-dependent.

Method

A four-test diagnostic protocol is proposed to audit claimed implicit computation in DEQs, identifying "lazy identity collapse" and gradient starvation by testing solver behavior.

In practice

• Apply the four-test diagnostic protocol to audit DEQ implicit computation.
• Beware of gradient starvation in DEQ training.
• Avoid relying solely on zeroing ablations for DEQ analysis.

Topics

• Deep Equilibrium Models
• Implicit Computation
• Gradient Starvation
• Model Diagnostics
• Neural Network Training

Code references

• locuslab/deq
• locuslab/mdeq

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• The Equilibrium Is the Initialization: Lazy Identity Collapse in Physics-Structured Deep Equilibrium Reasoning — Physics-structured Deep Equilibrium Models can exhibit "lazy identity collapse," where implicit computation is bypassed, yielding no performance benefit.
• Blackwell Approachability and Gradient Equilibrium are Equivalent — Gradient Equilibrium (GEQ) is algorithmically equivalent to Blackwell approachability, regret minimization, and calibration.
• Equilibrium Propagation and Hamiltonian Inference in the Diffusive Fitzhugh-Nagumo Model — Equilibrium Propagation and Hamiltonian Echo Backpropagation can be extended to skew-gradient systems and deep Energy-Based Models for efficient credit assignment and inference.
• Towards Understanding Gradient Flow Dynamics of Homogeneous Neural Networks Beyond the Origin — Gradient flow dynamics of homogeneous neural networks beyond the origin preserve sparsity and characterize saddle points.
• Empirical Minimal-Realisation Compression of Deep Neural Networks via Controllability-Observability Tests — Deep neural network compression can be achieved by applying controllability-observability tests to identify and reduce redundant internal states.
• Deep Unfolded Latent Optimally Partitioned-l2/l1 Networks for Data-driven Block-Sparse Recovery — Deep Unfolded LOP-l2/l1 networks overcome tuning and stability issues in block-sparse recovery using implicit differentiation or DWF.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.