Variance-reduced Domain Adaptation using Paired Sampling

2026-07-22 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Paired Sampling for Domain Adaptation (PSDA) is a novel stochastic variance reduction (SVR) technique designed to address high variance in distribution-matching losses like correlation alignment and maximum mean discrepancy, which often undermine unsupervised domain adaptation (UDA) effectiveness in minibatch optimisation. PSDA tackles the lack of finite-sum structure in these losses by pairing observations both within and across domains, forming quadruplets that are consistently sampled during training. This method is specifically engineered to minimize expected gradient variance, reducing to a set of linear assignment problems. Simulations confirm PSDA's reduced variance compared to related methods, and experiments across three domain shift datasets demonstrate improved target domain accuracy.

Key takeaway

For AI Scientists working on unsupervised domain adaptation, you should consider integrating Paired Sampling for Domain Adaptation (PSDA) to mitigate high variance in distribution-matching losses. This technique, which pairs observations to form quadruplets, can significantly improve target domain accuracy by reducing gradient variance during minibatch optimization. Implementing PSDA could lead to more robust and effective UDA models.

Key insights

PSDA reduces variance in UDA losses by pairing samples, improving accuracy on domain shift datasets.

Principles

• High variance in UDA losses undermines minibatch optimization.
• Pairing samples within and across domains minimizes gradient variance.
• Linear assignment problems can optimize sample pairings for SVR.

Method

PSDA forms quadruplets by pairing observations within and across domains, sampled together to minimize expected gradient variance, solvable via linear assignment problems.

In practice

• Apply PSDA to UDA tasks with high-variance losses.
• Consider paired sampling for gradient variance reduction.
• Utilize linear assignment for optimal sample grouping.

Topics

• Domain Adaptation
• Variance Reduction
• Stochastic Gradient Descent
• Unsupervised Learning
• Minibatch Optimization
• Paired Sampling

Best for: AI Scientist, Research Scientist

Related on AIssential

• Variance-reduced Domain Adaptation using Paired Sampling — PSDA reduces variance in UDA losses by pairing observations, improving target domain accuracy.
• Online Variance Reduction for Domain Adaptation on Streaming Data — ARROW is the first online SVR algorithm for MMD and CORAL, enabling domain adaptation on streaming data.
• How Useful is Causal Invariance for Domain Adaptation in Finite-Sample Settings? — Causal invariance improves supervised domain adaptation in finite-sample settings when target-risk margins between candidate models are sufficiently large.
• Variational Inference-Based Adversarial Domain Adaptation — VIADA combines VAEs and adversarial learning to overcome domain shift while preserving class structure.
• BP-TTA: Balanced and Prototype-Guided Test-Time Adaptation in Dynamic Scenarios — BP-TTA combines balanced sampling and prototype guidance to address class imbalance and continual domain shifts in dynamic test-time adaptation.
• Variance Reduction for Non-Log-Concave Sampling with Applications to Inverse Problems — Unified analysis shows variance reduction improves sampling from non-log-concave distributions, enhancing convergence and sample quality.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.