Partial Fusion of Neural Networks: Efficient Tradeoffs Between Ensembles and Weight Aggregation

2026-07-22 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

Partial Fusion of Neural Networks introduces a novel method to efficiently combine neural networks, interpolating between high-cost ensembles and less accurate weight aggregation. This technique, called partial fusion, merges only the most similar neurons across networks, offering a flexible tradeoff between computational cost and performance. The paper also presents generalized pruning, which extends traditional pruning by allowing neurons to be linearly combined, applicable to both ensembles and single-model compression. A specific implementation, Partial Optimal Transport (OT) Fusion, is detailed for partial fusion, alongside a clustering method for generalized pruning. Experiments on MLPs and CNNs (MNIST, CIFAR10) demonstrate that partial fusion provides efficient performance gains; for example, a 40% neuron increase via Partial OT Fusion significantly improved accuracy over standard OT fusion. Generalized pruning often yielded better tradeoffs, particularly for MLPs, despite higher computational costs. Fine-tuning partially fused models consistently outperformed individual models.

Key takeaway

For Machine Learning Engineers optimizing model deployment, this research offers a pathway to balance performance and computational overhead. You should consider partial fusion or generalized pruning to create aggregate models that achieve near-ensemble accuracy with significantly fewer parameters than full ensembles. Experiment with Partial OT Fusion for merging similar neurons or generalized pruning via clustering for flexible model compression, especially when fine-tuning with limited data. This approach can reduce inference costs while maintaining high predictive power.

Key insights

Partial fusion and generalized pruning offer flexible tradeoffs between neural network ensemble performance and computational cost.

Principles

• Neuron similarity differences are exploitable for model size/performance tradeoffs.
• Partial fusion acts as an inductive bias within generalized pruning.
• Layer-dependent treatment can improve model aggregation performance.

Method

Partial fusion uses partial optimal transport to match and average similar neurons, leaving others isolated. Generalized pruning uses clustering to combine or delete neurons.

In practice

• Use Partial OT Fusion for efficient ensemble-like performance.
• Apply generalized pruning for single-model compression.
• Fine-tune partially fused models with limited data for further gains.

Topics

• Neural Network Ensembles
• Weight Aggregation
• Model Compression
• Optimal Transport
• Generalized Pruning
• Partial Fusion

Code references

• Fabian-Mor/partial_fusion_nn

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• StackingNet: Collective Inference Across Independent AI Foundation Models — StackingNet coordinates independent black-box foundation models via meta-ensemble, improving accuracy, fairness, and robustness with minimal data.
• Neural Network Compression by Approximate Differential Equivalence — Compress neural networks by aggregating functionally similar neurons using differential equivalence, not just pruning weights.
• Compositional Boundaries for Density Fusion — Order-invariant distributed density fusion requires specific algebraic properties, often violated by divergence-based or approximate methods.
• Multiple Additive Neural Networks for Structured and Unstructured Data — MANN enhances gradient boosting with shallow neural networks for diverse data types, improving accuracy and robustness.
• Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions — Model merging offers a computationally efficient way to combine LLM capabilities without extensive retraining.
• Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning — Progressive magnitude-based pruning enables effective neural network sparsification within a single training cycle.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.