Sparse Inter-Layer Dependencies of Transformer FFN Neurons

2026-07-13 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

A new training-free attribution method reveals sparse inter-layer dependencies within Transformer Feedforward Network (FFN) blocks, which typically account for a large fraction of parameters and computation. This method estimates the relative influence of upstream neurons and attention outputs on a target neuron's activation. Empirically, the research found that only small subsets of preceding activations and attention outputs are sufficient to preserve neuron activations with high fidelity, even when other inputs are masked. This effective sparsity increases further when accounting for the inherent activation sparsity of upstream layers. Crucially, applying these neuron-specific masks across all layers simultaneously, allowing deviations to propagate, leaves model perplexity largely unchanged at moderate sparsity levels. These findings demonstrate that FFNs exhibit structured, sparse dependencies at the neuron level despite their dense parameterization, offering a scalable tool for circuit-level interpretability and identifying pathways for efficient inference.

Key takeaway

For Machine Learning Engineers optimizing Transformer models, understanding FFN internal sparsity is crucial. This research suggests you can achieve significant computational efficiency by identifying and exploiting sparse inter-layer dependencies without sacrificing model performance. Consider applying neuron-specific masking techniques to reduce FFN computation, potentially leading to faster inference and lower resource consumption for your deployments. This approach offers a path to more efficient and interpretable large language models.

Key insights

Transformer FFNs exhibit sparse, structured inter-layer dependencies at the neuron level, enabling circuit-level interpretability and efficient inference.

Principles

• FFN neuron activations depend on sparse upstream subsets.
• Masking non-influential inputs preserves activation fidelity.
• FFNs exhibit structured inter-layer dependencies.

Method

A training-free attribution method estimates the relative influence of upstream neurons and attention outputs on a target FFN neuron's activation. It identifies minimal input subsets preserving activation fidelity.

In practice

• Apply for circuit-level interpretability.
• Pinpoint sparse pathways for efficient inference.

Topics

• Transformer FFNs
• Neuron Sparsity
• Model Interpretability
• Efficient Inference
• Attribution Methods

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Sparse Inter-Layer Dependencies of Transformer FFN Neurons — Transformer FFN neurons exhibit sparse, structured inter-layer dependencies, discoverable via a training-free attribution method.
• Language Model Circuits Are Sparse in the Neuron Basis — MLP activations, combined with RelP attribution, offer a sparse and faithful basis for language model circuit tracing, comparable to SAEs.
• Sparse Autoencoders for Interpretable Out-of-Distribution Detection — Sparse autoencoders can learn distinct, interpretable features from intermediate layers, enabling robust out-of-distribution detection and interpretability.
• Resolving superposition in AI for interpretability and cross-modal alignment in patient-neuronal images — Sparse Autoencoders resolve neural network superposition, purifying latent space geometry for biological image interpretability.
• A Controlled Study of Attention-Only Transformers — Attention-only Transformers can match standard Transformer performance by reallocating feed-forward network parameters to increased attention depth.
• A Sharper Picture of Generalization in Transformers — Sparse Fourier spectra and low-sharpness constructions improve Transformer generalization on boolean domains.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.