One-Step Generalization Ratio Guided Optimization for Domain Generalization

2026-06-15 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

GENIE (Generalization-ENhancing Iterative Equalizer) is a novel optimizer designed to address domain generalization (DG) challenges by preventing models from overfitting to domain-specific features. Gradient-based DG methods often inadvertently reinforce spurious correlations, a problem GENIE tackles by leveraging the One-Step Generalization Ratio (OSGR). This ratio quantifies each parameter's contribution to loss reduction and assesses gradient alignment. By dynamically equalizing OSGR through a preconditioning factor, GENIE ensures that no small subset of parameters dominates optimization, thereby promoting the learning of domain-invariant features. Theoretically, GENIE balances convergence contribution and gradient alignment, achieving higher OSGR while maintaining SGD's convergence rate. Empirically, it outperforms existing optimizers and enhances performance when integrated with various DG and single-DG methods, published on 2026-06-15.

Key takeaway

For Machine Learning Engineers developing models for domain generalization, GENIE offers a promising approach to mitigate overfitting to domain-specific features. You should consider integrating GENIE into your training pipelines, especially when existing gradient-based methods struggle with spurious correlations. Its ability to balance parameter contributions and maintain SGD's convergence rate suggests it could improve model robustness and generalization to unseen target domains.

Key insights

GENIE optimizer uses One-Step Generalization Ratio (OSGR) to balance parameter contributions for robust domain generalization.

Principles

• Prevent a small subset of parameters from dominating optimization.
• Balance convergence contribution and gradient alignment among parameters.

Method

GENIE quantifies each parameter's contribution to loss reduction and gradient alignment using OSGR, then dynamically equalizes OSGR via a preconditioning factor.

In practice

• Integrates with various Domain Generalization methods.
• Enhances performance in single-DG scenarios.

Topics

• Domain Generalization
• Optimizer
• Machine Learning
• Gradient-based methods
• Parameter optimization
• OSGR

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Parameter Tuning with Generalization Guarantees for GPU-Accelerated Linear Programming — Data-driven algorithm design provides generalization guarantees for hyperparameter tuning in complex optimization solvers.
• Harmonizing Multi-Objective LLM Unlearning via Unified Domain Representation and Bidirectional Logit Distillation — Multi-objective LLM unlearning can be harmonized via unified data representation and bidirectional logit distillation.
• Separating Geometry from Probability in the Analysis of Generalization — Generalization can be analyzed deterministically via sensitivity of optimization solutions to data perturbations.
• Minibatch Selection via Partition Matroid Constrained Gradient Matching — PartitionSel optimizes LLM fine-tuning on heterogeneous data by balancing domain coverage and convergence via gradient-matching and matroid constraints.
• Training Neural Networks with Optimal Double-Bayesian Learning — A double-Bayesian framework theoretically derives optimal SGD hyperparameters, showing superior generalization over Adam.
• PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training — Polynomial preconditioning stabilizes LLM training by controlling weight singular values, incurring no inference overhead.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.