Sparse Inter-Layer Dependencies of Transformer FFN Neurons

2026-07-15 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

A new training-free attribution method reveals sparse inter-layer dependencies within Transformer Feedforward Network (FFN) blocks, which typically account for two-thirds of model parameters. This method estimates the influence of upstream neurons and attention outputs on a target neuron's activation. Empirical evaluation across GPT-2 variants and Qwen2.5-0.5B shows that small subsets of preceding activations and attention outputs, often 10% or more, preserve FFN neuron activations with high fidelity (AUROC scores above 0.9 at 5% sparsity). When neuron-specific masks are applied simultaneously across all layers, model perplexity remains largely unchanged at moderate sparsity levels. This indicates that despite dense parameterization, FFNs exhibit structured, sparse functional pathways at the neuron level, with effective sparsity compounding due to inherent activation sparsity.

Key takeaway

For AI Architects and Machine Learning Engineers optimizing Transformer models, understanding FFN's intrinsic sparse dependencies is crucial. You should consider this training-free attribution method to identify critical pathways, potentially guiding the design of more efficient, sparse-by-construction architectures or informing model distillation strategies. This approach offers a scalable tool for interpretability and could lead to compact models for faster inference without extensive retraining.

Key insights

Transformer FFN neurons exhibit sparse, structured inter-layer dependencies, discoverable via a training-free attribution method.

Principles

• FFN neuron activation depends on a sparse upstream subset.
• Dependencies show locality and layer-wise variability.
• Effective sparsity compounds with activation sparsity.

Method

Decompose FFN pre-activations into additive contributions, define relevance scores by squared dot product and activation variance, then causally test via mean-ablation activation patching.

In practice

• Identify sparse pathways for circuit-level analysis.
• Guide design of sparse-by-construction models.
• Inform model distillation for efficient inference.

Topics

• Transformers
• Feedforward Networks
• Mechanistic Interpretability
• Sparsity
• Attribution Methods
• Model Efficiency

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Sparse Inter-Layer Dependencies of Transformer FFN Neurons — Transformer FFNs exhibit sparse, structured inter-layer dependencies at the neuron level, enabling circuit-level interpretability and efficient inference.
• Engineering Verifiable Modularity in Transformers via Per-Layer Supervision — Architectural interventions and per-layer supervision can engineer verifiable modularity in Transformers, overcoming the "Hydra effect."
• A Sharper Picture of Generalization in Transformers — Sparse Fourier spectra and low-sharpness constructions improve Transformer generalization on boolean domains.
• A unifying framework from neural superposition to sparse interpretable codes — A framework unifies theories to extract interpretable, sparse features from neural network superposition.
• A Machine Learning Surrogate for Component Criticality Ranking in Interdependent Power-Communication Networks — A Gradient Boosting surrogate accurately predicts power-communication network component criticality, matching high-fidelity simulator reproducibility.
• Revisiting Neural Processes via Fourier Transform and Volterra Series — Combining Volterra series with frequency-domain convolutions yields scalable, interpretable translation-equivariant Neural Processes for irregular data.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.