Breaking Spurious Correlations via Generative Randomization and Cross-Variant Self-Supervised Learning

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

A new two-stage framework addresses deep neural network failures under distribution shifts caused by spurious correlations. This method, named "Breaking Spurious Correlations via Generative Randomization and Cross-Variant Self-Supervised Learning," first isolates foreground objects using zero-shot segmentation with Grounding DINO and SAM. It then generates diverse context-shifted variants by modifying backgrounds via a structure-preserving diffusion model, FLUX.1-Fill. The second stage introduces Cross-Variant Self-Supervised Learning, where variants of the same object with different backgrounds form positive pairs in a contrastive objective. The pretrained encoder is subsequently fine-tuned using an ERM warm-up followed by GroupDRO with layer-wise learning rates. This framework achieved 92.5% worst-group accuracy on Waterbirds, 81.7% on MetaShift, and 87.4% on NICO++.

Key takeaway

For machine learning engineers building robust vision systems, this framework offers a clear path to mitigate spurious correlations. By generating diverse background contexts and explicitly training for object-centric invariance, you can significantly improve model generalization under distribution shifts. Consider integrating generative randomization and cross-variant SSL into your training pipeline to enhance worst-group performance on critical benchmarks.

Key insights

Generative randomization and cross-variant self-supervised learning create background-invariant visual representations for robust models.

Principles

• Explicitly learn invariance across generated context shifts.
• GroupDRO benefits from an ERM warm-up phase.
• Layer-wise learning rates preserve invariant features during fine-tuning.

Method

Isolate objects with Grounding DINO/SAM. Generate context-shifted variants using FLUX.1-Fill. Apply Cross-Variant SSL with contrastive loss. Fine-tune with ERM warm-up and GroupDRO using layer-wise learning rates.

In practice

• Use diffusion models for semantic background randomization.
• Implement contrastive learning on generated context variants.
• Apply ERM warm-up before GroupDRO optimization.

Topics

• Spurious Correlations
• Distribution Shift
• Generative Randomization
• Cross-Variant SSL
• GroupDRO
• Diffusion Models

Code references

• surajyadav-research/GRSSL
• black-forest-labs/flux

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• TILDE: TILt-based Distributional Erasure for Concept Unlearning — Concept unlearning in diffusion models is best achieved by explicitly defining the desired post-unlearning distribution, not just suppressing concepts.
• Robust Representation Learning through Explicit Environment Modeling — Explicitly modeling and marginalizing environmental variation yields more robust OOD generalization than invariance-seeking methods.
• Robust training of implicit generative models for multivariate and heavy-tailed distributions with an invariant statistical loss — ISL offers a stable, gradient-based alternative to adversarial training for implicit generative models, enhanced for heavy-tailed and multivariate data.
• Towards Accurate and Calibrated Classification: Regularizing Cross-Entropy From A Generative Perspective — Generative Cross-Entropy improves DNN accuracy and calibration by regularizing class-level confidence from a generative perspective.
• Converge to Surprise: Evolutionary Self-supervised Image Clustering — Self-supervised image clustering can achieve state-of-the-art results by maximizing a "surprise score" without per-step loss targets, using an evolutionary strategy.
• Adversarial Label Invariant Graph Data Augmentations for Out-of-Distribution Generalization — RIA improves OoD generalization under covariate shift using adversarial label-invariant data augmentations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.