Semantically Aware Optimal Transport for Dense Label Transfer

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, medium

Summary

Semantically Aware Optimal Transport (SAOT) is a novel method for dense label transfer, designed to overcome the limitations of vision foundation models when transferring labels across domains with significant distribution shifts. SAOT learns a transport cost within a fused unbalanced optimal transport framework, leveraging a learnable appearance metric, semantic class-prototype priors, and a block-sparse solver for efficient inference. This core SAOT mechanism is paired with a two-stage decoder: an MLP initially trained on SAOT-generated pseudo-labels, subsequently refined through EMA-teacher self-training with class-balanced sampling. The SAOT+Decoder pipeline achieved a 25.7% mIoU on GTA5→Cityscapes using frozen DINOv2 ViT-L/14 features, representing a 3.8× improvement over the 6.7% mIoU of nearest-neighbor transfer. It also demonstrated strong performance on VOC train→val with frozen ViT-B/16 features, reaching 47.5% mIoU.

Key takeaway

For Machine Learning Engineers developing semantic segmentation models or tackling domain adaptation challenges, if you are experiencing poor label transfer performance across large distribution shifts with frozen vision foundation model features, SAOT provides a significant improvement. You should consider integrating its semantically aware optimal transport and two-stage decoder pipeline to achieve robust dense label transfer, as demonstrated by its 3.8× mIoU gain on synthetic-to-real benchmarks.

Key insights

SAOT improves dense label transfer across visual domains by learning semantic transport costs with unbalanced optimal transport.

Principles

• Semantic priors enhance cross-domain label transfer.
• Unbalanced optimal transport handles distribution shifts.
• Learned transport costs capture domain-invariant structure.

Method

SAOT learns transport costs using fused unbalanced optimal transport, combining appearance metrics and class-prototype priors. A two-stage decoder refines pseudo-labels via MLP training and EMA-teacher self-training with class-balanced sampling.

In practice

• Apply SAOT for synthetic-to-real domain adaptation.
• Use frozen ViT features for efficient label transfer.
• Implement a two-stage decoder for pseudo-label refinement.

Topics

• Dense Label Transfer
• Optimal Transport
• Vision Foundation Models
• Domain Adaptation
• Semantic Segmentation
• DINOv2 ViT

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Sample Complexity of Transfer Learning: An Optimal Transport Approach — Transfer learning significantly improves sample efficiency for complex models, especially with high data dimensions and non-smooth target models.
• Temporally Consistent Label Interpolation for Robust Surgical Multi-Task Learning under Challenging Conditions — FAROS uses flow-guided label interpolation to generate dense pseudo labels for robust surgical multi-task learning.
• ARIA: Adaptive Region-Based Importance Allocation for Conditional Diffusion Distillation — ARIA adaptively allocates training effort in conditional diffusion distillation by tracking teacher-student discrepancy across conditioning regions.
• Linearizing Vision Transformer with Test-Time Training — Linearizing Vision Transformers with Test-Time Training enables efficient weight transfer and faster inference.
• Inverse Entropic Optimal Transport Solves Semi-supervised Learning via Data Likelihood Maximization — Inverse entropic optimal transport provides a unified likelihood maximization framework for semi-supervised conditional distribution learning using mixed data.
• Vision Transformer-Conditioned UNet for Domain-Adaptive Semantic Segmentation — ViTC-UNet combines frozen ViT global priors with UNet local inductive biases for efficient, high-precision biomedical semantic segmentation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.