The JEPA Paradox in Language: The Geometry of Linguistic Alternatives

2026-07-26 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

Joint-Embedding Predictive Architectures (JEPAs), while effective for images, video, and audio, have not become a standard objective for text encoders. This is attributed to a fundamental mismatch between squared-error latent prediction and the conditional structure of language. Unlike the spatial continuity in local image prediction, masked text can admit multiple valid token or span completions whose representations do not necessarily share a coherent center. This mismatch, formalized through conditions like predictability and low conditional variance, creates centroid degeneracy and collapse pressure in text. Experiments with matched I-JEPA and T-JEPA models confirm this, revealing that mutual-information saturation and elevated target variance precede train--validation instability, effective-rank degeneration, cosine collapse, and poor downstream transfer. These findings, consistent across five independent data seeds, indicate that text-compatible JEPA objectives must preserve multiple plausible completions rather than compressing them into a single latent point.

Key takeaway

For AI Scientists and Machine Learning Engineers designing self-supervised learning objectives for text, recognize that standard Joint-Embedding Predictive Architectures (JEPAs) are fundamentally mismatched with language's conditional structure. Your models should avoid compressing multiple plausible text completions into a single latent point, as this leads to instability and poor downstream transfer. Instead, prioritize developing JEPA objectives that explicitly preserve the inherent ambiguity and diverse valid completions present in masked text.

Key insights

JEPAs struggle with text because language's conditional structure demands preserving multiple plausible completions, not compressing them into a single latent point.

Principles

• JEPAs need conditional concentration.
• Masked text has multiple valid completions.
• Mismatch causes centroid degeneracy.

Method

The mismatch is formalized via three conditions: predictability, non-collapse, and low conditional variance, which, when failed, lead to centroid degeneracy and collapse.

Topics

• Joint-Embedding Predictive Architectures
• Self-supervised Learning
• Text Encoders
• Latent Space Learning
• Natural Language Processing
• Representation Learning

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• How AI Learned to Teach Itself [JEPA] — Self-supervised learning evolves from contrastive methods to JEPA, predicting latent embeddings for robust world models.
• Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning — Combining JEPA latent prediction with MLM improves text encoder embedding uniformity and semantic-lexical balance without immediate linear-probe accuracy gains.
• Text-Conditional JEPA for Learning Semantically Rich Visual Representations — Text-Conditional JEPA uses image captions to reduce prediction uncertainty in masked feature prediction for richer visual representations.
• LatentWave: JEPA Pretraining for Wireless Foundation Models — JEPA pretraining for wireless foundation models enhances representation transferability by predicting masked regions in latent space.
• Identifiability Without Gaussianity: Symbolic World Models and Near-Infinite Temporal Consistency — Symbolic grounding in causal dynamics enables near-infinite temporal consistency and exact identifiability in world models, overcoming Gaussianity limits.
• BRo-JEPA: Learning Modular Arithmetic in Latent Space — Latent world models can learn symbolic rules if their architecture aligns with the problem's inherent structure.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.