VTaMo: Video-Text Alignment Model for Sign Language Translation

2026-07-13 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

VTaMo is a novel framework for gloss-free Sign Language Translation (SLT) that explicitly addresses cross-modal alignment challenges. It introduces multi-granularity alignment at three levels: local alignment using entropy-regularized optimal transport with a learnable null token for fine-grained frame-to-token correspondences; global alignment via a learnable orthogonal transformation to calibrate embedding space geometry through Earth Mover's Distance; and position-aligned contrastive learning for discriminative token-level representations. Utilizing a CLIP-ViT-Large visual encoder and a LoRA-adapted Flan-T5-XL language model, VTaMo achieves state-of-the-art performance across four benchmarks: Phoenix-2014T (28.86 BLEU-4), CSL-Daily (27.16 BLEU-4), How2Sign (18.47 BLEU-4, 50.26 BLEURT), and OpenASL (25.94 BLEU-4, 62.48 BLEURT). The framework reorders visual features during training to match target pseudo-gloss token order, with a text-only recovery model handling spoken language order at inference. Ablation studies confirm each alignment component's complementary contribution and the model's robustness to appearance shifts.

Key takeaway

For Machine Learning Engineers developing gloss-free Sign Language Translation systems, VTaMo demonstrates that explicitly modeling multi-granularity vision-text alignment is crucial. You should integrate local optimal transport with null tokens, global orthogonal transformations, and position-aligned contrastive learning to resolve semantic and temporal mismatches. This approach significantly boosts translation accuracy, especially on diverse, large-vocabulary datasets, and allows for robust performance without visual fine-tuning. Consider adopting this explicit alignment strategy to improve your model's ability to handle complex sign language structures.

Key insights

Explicit multi-granularity alignment significantly improves gloss-free sign language translation by resolving vision-text semantic and temporal mismatches.

Principles

• Sign language translation benefits from explicit frame-to-token alignment.
• Orthogonal transformations can align embedding spaces without distortion.
• Null tokens handle non-lexical visual segments in continuous signing.

Method

VTaMo employs entropy-regularized optimal transport for local frame-to-token alignment, a learnable orthogonal transformation for global sentence-level alignment, and position-aligned contrastive learning, followed by window-based feature reordering for decoder input.

In practice

• Use pseudo-gloss to simplify alignment targets.
• Employ epsilon annealing for stable optimal transport.
• Train a text-only model for spoken word order recovery.

Topics

• Sign Language Translation
• Cross-Modal Alignment
• Optimal Transport
• Contrastive Learning
• Gloss-Free SLT
• Flan-T5

Code references

• junyi2005/vtamo

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• VTaMo: Video-Text Alignment Model for Sign Language Translation — VTaMo explicitly aligns video frames and text tokens at multiple granularities for improved sign language translation.
• Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Field — Consistent evaluation reveals many reported SLT performance gains are artifacts of inconsistent implementation and evaluation setups.
• Target-Side Paraphrase Augmentation for Sign Language Translation with Large Language Models — LLM-generated target-side paraphrases can augment Sign Language Translation, improving semantic fidelity, but effectiveness varies by corpus characteristics.
• M3T: Discrete Multi-Modal Motion Tokens for Sign Language Production — Integrating high-dimensional facial models and collapse-free tokenization is crucial for linguistically complete sign language production.
• Simultaneous Speech-to-Speech Translation Without Aligned Data — Simultaneous S2ST/S2TT can achieve state-of-the-art performance without word-level aligned data, using RL to optimize latency.
• V-LynX: Token Interface Alignment for Video+X LLMs — Video LLMs possess a continuous "token interface" allowing visual tokens to operate independently, which V-LynX leverages for efficient multimodal integration.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.