Variance-reduced Domain Adaptation using Paired Sampling

2026-07-22 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Andrea Napoli introduces Paired Sampling for Domain Adaptation (PSDA), a novel stochastic variance reduction (SVR) technique designed to address high variance issues in unsupervised domain adaptation (UDA). Existing distribution-matching frameworks like correlation alignment and maximum mean discrepancy (MMD) suffer from high variance in minibatch optimization, hindering their effectiveness. Furthermore, these losses lack the finite-sum structure required by classical SVR methods. PSDA tackles this by pairing observations both within and across domains, forming quadruplets that are consistently sampled during training. These pairings are specifically designed to minimize expected gradient variance and are determined by solving a set of linear assignment problems. Simulations confirm PSDA's reduced variance compared to related methods, and experiments across three distinct domain shift datasets demonstrate improved target domain accuracy.

Key takeaway

For Machine Learning Engineers developing unsupervised domain adaptation solutions, if you are encountering high variance in correlation alignment or MMD losses during minibatch training, you should consider implementing Paired Sampling for Domain Adaptation (PSDA). This technique directly addresses gradient variance, potentially improving your model's target domain accuracy on domain shift datasets. Evaluate PSDA's impact on your specific UDA tasks to optimize performance and stability.

Key insights

PSDA reduces variance in UDA losses by pairing observations, improving target domain accuracy.

Principles

• High variance in UDA losses undermines minibatch optimization.
• Finite-sum structure is critical for classical SVR methods.
• Strategic observation pairing can minimize gradient variance.

Method

PSDA pairs observations within and across domains to form quadruplets, sampled together. Pairings minimize expected gradient variance by solving linear assignment problems.

In practice

• Apply PSDA to UDA tasks using correlation alignment or MMD.
• Implement linear assignment problems for optimal observation pairing.
• Evaluate PSDA on domain shift datasets for accuracy gains.

Topics

• Unsupervised Domain Adaptation
• Stochastic Variance Reduction
• Correlation Alignment
• Maximum Mean Discrepancy
• Paired Sampling
• Minibatch Optimization

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Variance-reduced Domain Adaptation using Paired Sampling — PSDA reduces variance in UDA losses by pairing samples, improving accuracy on domain shift datasets.
• How Useful is Causal Invariance for Domain Adaptation in Finite-Sample Settings? — Causal invariance improves supervised domain adaptation in finite-sample settings when target-risk margins between candidate models are sufficiently large.
• Online Variance Reduction for Domain Adaptation on Streaming Data — ARROW is the first online SVR algorithm for MMD and CORAL, enabling domain adaptation on streaming data.
• Safe, Scalable, and Accurate Bayes Posterior Sampling for Large-Data Generalized Linear Mixed Models — SMLD offers a safe, scalable, and accurate Bayesian sampling method for large-data GLMMs with constrained parameters.
• Variance Reduction for Non-Log-Concave Sampling with Applications to Inverse Problems — Unified analysis shows variance reduction improves sampling from non-log-concave distributions, enhancing convergence and sample quality.
• Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation — Deep Embedded Validation (DEV) provides an unbiased, bounded-variance method for model selection in Deep UDA.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.