From Perturbation Correction to Geometry-Aware Sampling: Sharpness-Guided Equilibrium Sampling for Balanced Flat Minima in Long-Tailed Learning

2026-07-24 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Sharpness-Guided Equilibrium Sampling (SGS) is a novel approach addressing poor generalization in long-tailed learning, where head classes dominate training and under-represented classes converge to sharper loss landscape regions. Unlike conventional re-sampling or existing long-tailed sharpness-aware minimization (SAM) methods that modify losses or perturbations after mini-batch drawing, SGS treats the sampling distribution as an active control variable for optimization geometry. It dynamically adjusts subsequent mini-batches by increasing sampling probability for less frequently sampled classes and suppressing those with large SAM-induced loss changes. SGS uses cumulative class counts and EMA sharpness estimates from standard SAM updates, avoiding class-wise perturbations or additional backward passes. This method is characterized by a continuous-time stochastic differential equation and a sampling-dependent PAC-Bayes analysis, demonstrating how frequency-sharpness feedback balances flatness. SGS-SAM improves Focal-SAM by 10.85 points in tail accuracy and 3.56 points overall on CIFAR-100 LT with an imbalance ratio of 100. On ImageNet-LT, it boosts ImbSAM by 6.59 points on tail classes and 1.20 points overall, with training time only \$1.02\times$ that of vanilla SAM.

Key takeaway

For Machine Learning Engineers optimizing models in long-tailed data scenarios, you should consider integrating Sharpness-Guided Equilibrium Sampling (SGS) to improve generalization. This method offers a novel sampling-side route to control loss-landscape geometry, significantly boosting tail class accuracy (e.g., 10.85 points on CIFAR-100 LT) with minimal training overhead (\$1.02\times$ vanilla SAM). You can achieve more balanced flatness profiles by dynamically adjusting class sampling based on frequency and sharpness, rather than solely relying on loss modifications.

Key insights

Sharpness-Guided Equilibrium Sampling (SGS) dynamically adjusts class sampling to achieve balanced flat minima in long-tailed learning.

Principles

• Sampling distribution can control optimization geometry.
• Frequency-sharpness feedback balances flatness.
• Jointly regulate data exposure and optimization geometry.

Method

SGS dynamically adjusts mini-batches by increasing sampling for less frequent classes and suppressing those with large SAM-induced loss changes, using cumulative class counts and EMA sharpness estimates.

In practice

• Integrate SGS with existing SAM methods.
• Apply to long-tailed image classification.
• Explore sampling as a loss-landscape control.

Topics

• Long-tailed Learning
• Sharpness-Aware Minimization
• Geometry-Aware Sampling
• Loss Landscape Control
• Model Generalization
• Imbalanced Data

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• A Function-Centric Perspective on Flat and Sharp Minima — Sharpness in neural network minima is function-dependent, not solely indicative of poor generalization.
• Accelerating Constrained Sampling: A Large Deviations Approach — A specific skew-symmetric matrix design for SRNLD accelerates constrained sampling and reduces asymptotic variance.
• Dynamic Cluster Data Sampling for Efficient and Long-Tail-Aware Vision-Language Pre-training — Dynamic cluster-based sampling efficiently trains VLMs, improving long-tail concept performance by balancing data distribution.
• Curvature-Aligned Probing for Local Loss-Landscape Stabilization — Curvature-aligned probing in top Hessian eigenspaces offers a more efficient and accurate way to assess loss-landscape stabilization.
• REMAL: Residual Equilibrium Manifold Active Learning for Surrogate-Based Multidisciplinary Design Analysis — REMAL models the residual equilibrium manifold using multitask Gaussian processes and active learning for efficient fixed-point prediction in coupled engineering systems.
• Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization — S-Adam stabilizes non-smooth deep learning optimization by dynamically adjusting step sizes based on local geometric instability, preventing gradient chattering.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.