SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport

2026-02-26 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

SOTAlign is a two-stage framework designed for semi-supervised alignment of unimodal vision and language models, addressing the challenge of achieving meaningful alignment with substantially less supervision than typical methods. It operates by first establishing a coarse shared geometry using a linear teacher on a small number of image-text pairs. Subsequently, SOTAlign refines this alignment on large amounts of unpaired data through an optimal-transport-based divergence, which transfers relational structure without overconstraining the target space. This approach allows SOTAlign to effectively leverage both paired and unpaired samples, learning robust joint embeddings across various datasets and encoder pairs. The framework significantly outperforms existing supervised and semi-supervised baselines, demonstrating its efficacy in scenarios with limited paired data.

Key takeaway

For research scientists developing multimodal AI systems, SOTAlign offers a compelling approach to model alignment, particularly when high-quality paired image-text data is scarce. You should consider integrating optimal transport-based divergence methods to leverage abundant unpaired data, potentially reducing annotation costs and improving cross-modal understanding in resource-constrained settings.

Key insights

SOTAlign aligns unimodal vision and language models using limited paired data and abundant unpaired data via optimal transport.

Principles

• Neural networks converge to shared world models.
• Optimal transport can transfer relational structure.

Method

SOTAlign uses a two-stage process: first, a linear teacher recovers coarse geometry from paired data; then, optimal transport refines alignment using unpaired samples.

In practice

• Align models with few paired samples.
• Utilize large unpaired datasets effectively.

Topics

• SOTAlign
• Semi-Supervised Learning
• Vision-Language Models
• Optimal Transport
• Model Alignment

Best for: Research Scientist, AI Researcher, AI Scientist, AI Student

Related on AIssential

• Reinforcing Dual-Path Reasoning in Spatial Vision Language Models — SR-REAL improves spatial VLMs by combining linguistic deduction and 3D geometric grounding via reinforcement learning for robust, generalizable reasoning.
• Inverse Entropic Optimal Transport Solves Semi-supervised Learning via Data Likelihood Maximization — Inverse entropic optimal transport provides a unified likelihood maximization framework for semi-supervised conditional distribution learning using mixed data.
• Mind the Heads: Topological Representation Alignment for Multimodal LLMs — HeRA improves MLLMs by aligning individual attention heads based on topological structure, surprisingly focusing on the least aligned.
• Language Acquisition Device in Large Language Models — LAD-inspired pre-pretraining with MP-Struct improves LLM data efficiency and instills human-like linguistic biases.
• SRA: Span Representation Alignment for Large Language Model Distillation — Aligning tokenizer-agnostic spans via their attention-weighted Center of Mass improves cross-tokenizer knowledge distillation.
• Uncovering Symmetry Transfer in Large Language Models via Layer-Peeled Optimization — LLM training implicitly transfers data symmetries into structured geometric configurations within model parameters.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.