Symmetrization of Loss Functions for Robust Training of Neural Networks in the Presence of Noisy Labels

2026-05-21 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new study introduces a symmetrization method for robust training of neural networks, addressing the challenge of noisy labels in expensive and error-prone training sets. This method arises from the unique decomposition of any multi-class loss function into a symmetric component and a class-insensitive term. Specifically, symmetrizing the cross-entropy loss results in a linear multi-class extension of the unhinged loss. Under suitable assumptions, this multi-class unhinged loss is demonstrated to be the unique convex multi-class symmetric loss. Furthermore, it plays a fundamental local role, as the linear approximation of any symmetric loss around score vectors with equal components is equivalent to it. The researchers also introduce SGCE and alpha-MAE, two novel loss functions that interpolate between the multi-class unhinged loss and the Mean Absolute Error, offering control over the beta-smoothness of the loss. Experiments on standard noisy-label benchmarks confirm their competitive performance against existing robust loss functions.

Key takeaway

For Machine Learning Engineers training neural networks with datasets prone to noisy labels, you should evaluate the newly introduced SGCE and alpha-MAE loss functions. These functions, derived from a principled symmetrization of cross-entropy, offer a robust alternative to existing methods. Their ability to interpolate between the multi-class unhinged loss and Mean Absolute Error, while controlling beta-smoothness, provides flexibility. Incorporating them could significantly improve model robustness and performance in real-world, imperfect data scenarios.

Key insights

Symmetrizing multi-class loss functions, especially cross-entropy, yields a unique convex unhinged loss robust to noisy labels.

Principles

• Symmetry offers theoretical robustness guarantees.
• Multi-class losses uniquely decompose into symmetric and class-insensitive parts.
• The multi-class unhinged loss is uniquely convex and symmetric.

Method

Decompose multi-class loss functions into symmetric and class-insensitive terms. Symmetrize cross-entropy to derive a multi-class unhinged loss. Introduce SGCE and alpha-MAE to interpolate between this unhinged loss and MAE, controlling beta-smoothness.

In practice

• Apply SGCE or alpha-MAE for robust training with noisy labels.
• Consider multi-class unhinged loss for convex symmetric loss needs.

Topics

• Loss Functions
• Noisy Labels
• Neural Network Training
• Symmetrization
• Cross-Entropy
• SGCE
• alpha-MAE

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• Loss Function Explained For Noobs (How Models Know They Are Wrong) — A loss function quantifies model prediction errors, guiding iterative adjustments to improve accuracy.
• Expressivity of congruence-based architectures for DNNs on positive-definite matrices — Semi-orthogonality in congruence layers limits DNN expressivity for SPD matrix classification due to spectral diversity loss.
• Towards Accurate and Calibrated Classification: Regularizing Cross-Entropy From A Generative Perspective — Generative Cross-Entropy improves DNN accuracy and calibration by regularizing class-level confidence from a generative perspective.
• Distributional Loss for Robust Classification — Distributional Loss uses a bimodal Gaussian distribution as a softer target for classifier outputs, enhancing robustness and mitigating overfitting without extra labels.
• The Matching Principle: A Geometric Theory of Loss Functions for Nuisance-Robust Representation Learning — Robustness techniques converge on estimating and regularizing encoder Jacobians against label-preserving deployment nuisance covariance.
• Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networks — SGD noise provides information about feature learning progression in deep linear networks without altering saddle-to-saddle dynamics.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.