Do We Need to Penalize Variance of Losses for Learning with Label Noise?

2025-12-31 · Source: JMLR · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, quick

Summary

The paper by Yexiong Lin, Yu Yao, Yuxuan Du, Jun Yu, Bo Han, Mingming Gong, and Tongliang Liu, published in 2026, investigates the role of loss variance in algorithms designed for learning with noisy labels. While conventional understanding suggests penalizing loss variance to reduce estimation error and discrepancy between average and expected risk, this research presents a counter-intuitive finding. The authors discovered that, for label-noise learning, encouraging a large variance of losses is actually beneficial. This approach effectively boosts the "memorization effect" and significantly reduces the detrimental impact of incorrect labels. The study demonstrates that simple regularizers can be designed to promote this large loss variance, easily integrating into many existing algorithms. Empirically, their proposed method consistently improved the generalization ability of baseline models across both synthetic and real-world datasets.

Key takeaway

For Machine Learning Engineers developing models for noisy datasets, you should reconsider the conventional approach of penalizing loss variance. Instead, actively encouraging a large variance of losses can significantly boost your model's memorization effect and reduce the harmfulness of incorrect labels. Implement regularizers designed to promote this large variance, as they can be easily integrated into existing algorithms to improve generalization on both synthetic and real-world data.

Key insights

For label-noise learning, encouraging a large variance of losses boosts memorization and reduces the harm of incorrect labels.

Principles

• Conventional loss variance principles may not apply to noisy labels.
• Boosting memorization can mitigate label noise impact.
• Loss variance is a controllable factor for noise robustness.

Method

Design regularizers that actively encourage a large variance of losses. These can be easily integrated into many existing statistically consistent algorithms to improve generalization with noisy labels.

In practice

• Integrate variance-encouraging regularizers into existing models.
• Test models with increased loss variance on noisy datasets.

Topics

• Label Noise Learning
• Loss Variance
• Regularization
• Memorization Effect
• Noisy Datasets
• Model Robustness

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• Regularisation in Neural Networks — Regularization techniques are crucial for neural networks to prevent overfitting and improve generalization on unseen data.
• Rethinking Noise-Robust Training for Frozen Vision Foundation Models: A Cross-Dataset Benchmark with a Case Study of Small-Loss Failure — No universal noisy-label learning method exists for frozen Vision Foundation Models; selection must be regime-aware.
• Issue #132 - Bagging: Why Training on Broken Copies of Your Data Works — Bagging reduces model variance by averaging predictions from models trained on bootstrapped data, offering free out-of-bag validation.
• 20+ Types of Loss Functions in Machine Learning — Selecting the appropriate loss function is crucial for effective model training across diverse machine learning tasks.
• The Implicit Bias of Logit Regularization — Logit regularization shifts classifier optimization from margin maximization to logit clustering, aligning weights with Fisher's Linear Discriminant.
• Cram Less to Fit More: Training Data Pruning Improves Memorization of Facts — Fact memorization in LLMs is limited by model capacity and exacerbated by skewed training data distributions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by JMLR.