Text-Conditional JEPA for Learning Semantically Rich Visual Representations

2026-05-07 · Source: Apple Machine Learning Research · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Text-Conditional JEPA (TC-JEPA) introduces a novel approach to visual self-supervised learning by integrating image captions to mitigate prediction uncertainty in masked feature prediction. Building upon the Image-based Joint-Embedding Predictive Architecture (I-JEPA), TC-JEPA employs a fine-grained text conditioner that utilizes sparse cross-attention over input text tokens to modulate predicted patch features. This conditioning makes patch features more predictable as a function of text, leading to more semantically meaningful visual representations. The proposed TC-JEPA demonstrates improved downstream performance and enhanced training stability, exhibiting promising scaling properties. Furthermore, it establishes a new vision-language pretraining paradigm based solely on feature prediction, surpassing contrastive methods on various tasks, particularly those demanding fine-grained visual understanding and reasoning.

Key takeaway

For research scientists developing self-supervised learning models, TC-JEPA suggests that integrating text conditioning can significantly enhance the semantic richness and predictability of visual representations. You should consider incorporating fine-grained text conditioners in your feature prediction architectures, especially for tasks requiring detailed visual understanding, as this approach has shown to outperform contrastive methods and improve training stability.

Key insights

Text-Conditional JEPA uses image captions to reduce prediction uncertainty in masked feature prediction for richer visual representations.

Principles

• Text conditioning improves visual feature predictability.
• Feature prediction can outperform contrastive methods.

Method

TC-JEPA modulates predicted patch features with a fine-grained text conditioner, computing sparse cross-attention over input text tokens to reduce visual uncertainty.

In practice

• Apply text conditioning to improve masked feature prediction.
• Explore feature prediction for vision-language pretraining.

Topics

• Text-Conditional JEPA
• Joint-Embedding Predictive Architecture
• Visual Self-Supervised Learning
• Vision-Language Pretraining
• Masked Feature Prediction

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• How AI Learned to Teach Itself [JEPA] — Self-supervised learning evolves from contrastive methods to JEPA, predicting latent embeddings for robust world models.
• Enhancing Visual Representation with Textual Semantics: Textual Semantics-Powered Prototypes for Heterogeneous Federated Learning — Textual semantics from LLMs and PLMs can significantly enhance prototype quality and model generalization in heterogeneous federated learning.
• Multi-Label Test-Time Adaptation with Bayesian Conditional Priors — BCP enhances multi-label VLM recognition under distribution shift by injecting label dependency via anchor-conditioned Bayesian refinement.
• Correlation Is Not Enough: Embedding Human Metadata for Individual Causal Discovery — Pretrained biomedical language models struggle with cross-domain causal inference due to false embedding proximity, requiring specialized contrastive training.
• From Alignment to Prediction: A Study of Self-Supervised Learning and Predictive Representation Learning — Predictive Representation Learning (PRL) offers a new self-supervised paradigm for predicting unobserved data components.
• Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning — Combining JEPA latent prediction with MLM improves text encoder embedding uniformity and semantic-lexical balance without immediate linear-probe accuracy gains.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Apple Machine Learning Research.