Compositionality Emerges in a Narrow Depth-Connectivity Regime: Architecture Constraints and Solution Manifolds

2026-06-18 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

This work reveals that compositionality, crucial for generalization in neural networks, emerges within a specific "narrow connectivity-depth sweet spot." Compositionality is observed only in certain sparse networks, where the specific connections retained are more critical than overall sparsity. Along the depth axis, it peaks within a narrow, target-dependent range, with both shallower and deeper networks failing. When these architectural conditions are not met, gradient descent yields fractured, non-compositional solutions. To address this, the authors introduce similarity-based pruning (SP) to achieve compositional connectivity and a heuristic depth predictor to estimate optimal depth. These empirical findings are supported by a theoretical framework involving compositional sparsity and feature-interference bounds.

Key takeaway

For AI Scientists designing neural network architectures, understanding that compositionality is not a given but an emergent property of specific depth and connectivity regimes is critical. You should experiment with similarity-based pruning and heuristic depth predictors to identify the "sweet spot" for your target task, avoiding architectures that silently converge to fractured solutions. Prioritize specific connection patterns over general sparsity to enhance generalization.

Key insights

Compositionality in neural networks arises only within a precise, narrow regime of network depth and connectivity.

Principles

• Compositionality requires specific sparse network connections.
• Optimal depth for compositionality is target-dependent.
• Violating depth/connectivity leads to fractured solutions.

Method

The paper introduces similarity-based pruning (SP) to recover compositional connectivity and a heuristic depth predictor to estimate optimal compositional depth.

In practice

• Apply SP for targeted network sparsity.
• Utilize depth predictor for architecture design.
• Focus on specific connection patterns, not just overall sparsity.

Topics

• Neural Network Compositionality
• Network Sparsity
• Deep Learning Architecture
• Similarity-based Pruning
• Generalization
• Gradient Descent

Best for: Research Scientist, AI Scientist

Related on AIssential

• Guided learning lets “untrainable” neural networks realize their potential — Brief representational alignment can enable "untrainable" neural networks to learn effectively by transferring architectural biases.
• Compositional Boundaries for Density Fusion — Order-invariant hierarchical execution of density fusion rules has specific compositional boundaries.
• Universality of Shallow and Deep Neural Networks on Non-Euclidean Spaces — TFNNs extend universal approximation to non-Euclidean spaces, even with width constraints, by leveraging topological properties.
• The Information-Theoretic Benefit of Shared Representations under Orthogonality Constraints — Joint approximation significantly reduces description-length in multi-task deep learning when tasks share a latent hard feature.
• The "Final Boss" of Deep Learning — Intrinsic architectural improvements, not just tool use, are crucial for robust algorithmic reasoning in language models.
• A Compositional Framework for Open-ended Intelligence — Open-ended intelligence is formalized as a compositional closure of primitives and operators, enabling unbounded adaptive generation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.