Contrast encodes inductive bias: separating slow noise from dynamics in predictive representation learning

2026-06-05 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A recent study identifies a critical failure mode in self-supervised methods like JEPA that learn representations and predict dynamics in latent space. These methods, particularly those employing contrastive predictive objectives that sample negatives across trajectories, often confuse slowly varying noise with the true dynamical signals. When noise features remain constant within a trajectory, the objective preferentially encodes this noise, leading to representations dominated by trajectory-specific noise. This degrades downstream performance, even with increased training data. The research demonstrates this issue and its remedy using a SimCLR-style JEPA on a synthetic moving-dot dataset and DySIB on rigid-body pendulum movies. The proposed solution involves sampling negatives within a single trajectory, which eliminates the predictive shortcut and forces the encoder to learn variables relevant for dynamics, improving representation quality even with strong slow noise.

Key takeaway

For Machine Learning Engineers developing self-supervised methods for dynamic systems, especially with noisy observations, you should critically re-evaluate your contrastive predictive objectives. If your current approach samples negative examples across trajectories, consider switching to within-trajectory negative sampling. This modification prevents the model from confusing slow noise with true dynamics, leading to more robust and accurate representations. Prioritize longer training trajectories to further enhance representation quality, even when dealing with strong slow noise in your experimental data.

Key insights

Contrastive predictive objectives fail to separate slow noise from dynamics when negatives are sampled across trajectories.

Principles

• Sampling negatives across trajectories encodes slow noise.
• Within-trajectory negative sampling isolates true dynamics.
• Longer trajectories improve dynamic representation learning.

Method

Modify contrastive predictive objectives to sample negative examples exclusively within a single trajectory, rather than across multiple trajectories, to prevent encoding slow noise.

In practice

• Implement within-trajectory negative sampling.
• Design experiments with longer training trajectories.
• Evaluate representation robustness to slow noise.

Topics

• Self-supervised Learning
• Contrastive Learning
• Representation Learning
• Dynamical Systems
• Noise Separation
• JEPA

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Generalization of Diffusion Models Arises with a Balanced Representation Space — Diffusion model generalization and memorization are directly linked to the balance and spikiness of their internal representations.
• Colored Noise Diffusion Sampling — Diffusion model SDE inference can be optimized by dynamically allocating noise energy based on frequency resolution.
• The Matching Principle: A Geometric Theory of Loss Functions for Nuisance-Robust Representation Learning — Robustness techniques converge on estimating and regularizing encoder Jacobians against label-preserving deployment nuisance covariance.
• Predictive Coding with Bayesian Priors via Proximal Gradients — Predictive coding is reinterpreted as continuous-time proximal gradient descent on a regularized maximum-a-posteriori objective.
• Domain-Aware Hierarchical Contrastive Learning for Semi-Supervised Generalization Fault Diagnosis — DAHCL improves semi-supervised fault diagnosis by correcting pseudo-label bias and leveraging uncertain unlabeled data through domain-aware and hierarchical contrastive learning.
• Towards Understanding Gradient Flow Dynamics of Homogeneous Neural Networks Beyond the Origin — Gradient flow dynamics of homogeneous neural networks beyond the origin preserve sparsity and characterize saddle points.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.