Label-Noise Resistant Learning via Optimal Brain Damage Masking

2026-07-23 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

A novel Dynamic Connection Masking (DCM) mechanism is proposed for deep neural networks (DNNs) to combat noisy labels, applicable to both Multi-Layer Perceptron Networks (MLPs) and Kolmogorov-Arnold Networks (KANs). DCM adaptively masks less important edges during training by evaluating their information-carrying capacity, which theoretically reduces gradient error. Extensive experiments on synthetic datasets like CIFAR-10, CIFAR-100, and CIFAR80-NO, as well as real-world datasets such as WebVision-Mini and Clothing1M, demonstrate that DCM consistently outperforms state-of-the-art (SOTA) approaches. For instance, ANL-DMLP and ANL-DKAN improved ANL by 3.42% and 3.84% on CIFAR-10 under 40% asymmetric noise. DISC-DKAN achieved 81.00% on WebVision-Mini and 74.49% on Clothing1M. The study also reveals KANs' superior noise robustness over MLPs in real-world scenarios, and DCM is a plug-and-play module that integrates seamlessly with existing noise-robust training methods.

Key takeaway

For Machine Learning Engineers developing robust classifiers, consider integrating Dynamic Connection Masking (DCM) into your training pipelines. This plug-and-play mechanism, especially with Kolmogorov-Arnold Networks (KANs), significantly improves noise robustness by adaptively pruning less important connections. You should evaluate DCM with your chosen robust loss functions or sample selection strategies, aiming for optimal masking ratios (e.g., p=0.6) to mitigate performance degradation from noisy labels.

Key insights

Dynamic Connection Masking (DCM) enhances deep network robustness to noisy labels by adaptively pruning less important connections during training.

Principles

• Masking less important edges reduces noisy gradient backpropagation.
• KANs inherently offer superior noise robustness over MLPs.
• Dynamic architectural regularization improves noise tolerance.

Method

DCM involves two steps: (i) calculating edge importance scores using standard deviation of activation variability, and (ii) dynamically masking edges with lower scores based on a hyperparameter p (masking ratio) at each training step.

In practice

• Integrate DCM with existing robust loss functions like NCE+RCE or ANL.
• Combine DCM with sample selection strategies such as DISC or SED.
• Apply node-wise masking at each training iteration for better performance.

Topics

• Dynamic Connection Masking
• Label Noise Robustness
• Kolmogorov-Arnold Networks
• Multi-Layer Perceptrons
• Gradient Error Reduction
• Architectural Regularization

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Student

Related on AIssential

• Rethinking Noise-Robust Training for Frozen Vision Foundation Models: A Cross-Dataset Benchmark with a Case Study of Small-Loss Failure — No universal noisy-label learning method exists for frozen Vision Foundation Models; selection must be regime-aware.
• A Practitioner's Guide to Kolmogorov-Arnold Networks — KANs offer enhanced expressivity and interpretability over MLPs by using learnable univariate basis functions on network edges.
• MSA-DCNN: A Data-Efficient Multi-Scale Deformable CNN for Medical Image Classification — MSA-DCNN unifies adaptive sampling, multi-scale attention, and self-distillation for data-efficient medical image classification.
• An Adaptive Data cleaning Framework for Noisy Label Detection — Multi-metric clustering on a unified feature space adaptively detects noisy labels without manual thresholds or noise priors.
• Adversarial Defence without Adversarial Defence: Enhancing Language Model Robustness via Instance-level Principal Component Removal — Transforming PLM embedding spaces to approximate Gaussian properties enhances adversarial robustness without costly adversarial training.
• Conformal Margin Risk Minimization: An Envelope Framework for Robust Learning under Label Noise — CMRM improves noisy label learning by using a quantile-calibrated regularization term without privileged knowledge.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.