Similarity search generalisation in contrastive learning with InfoNCE loss

2026-07-10 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new analysis of the InfoNCE loss function, widely used in contrastive learning for embedding models in similarity search, reveals that its population risk with k negative samples is O(1/k) close to an expected cross-entropy. This cross-entropy measures the divergence between a softmax similarity search on unseen data using learned embeddings and an idealised search based on the positive sample generator's implicit similarity. This interpretation extends existing views focused on mutual information or embedding alignment and uniformity. To quantify generalization, the research introduces a novel continuity bound for InfoNCE loss, derived through Gâteaux differentiation. This bound maintains the averaging structure over negative samples and includes an "inverse temperature" parameter for algorithmic temperature tuning. For Lipschitz embedding functions, the study demonstrates that increasing k negative samples stabilizes the generalization error.

Key takeaway

For research scientists optimizing contrastive learning systems with InfoNCE loss, understanding the O(1/k) population risk provides a clearer path to improving similarity search. You should consider increasing the number of k negative samples to stabilize generalization error, especially when working with Lipschitz embedding functions. Additionally, tune the "inverse temperature" parameter to account for algorithmic temperature, refining your model's performance on unseen data.

Key insights

InfoNCE loss's population risk is O(1/k) close to an expected cross-entropy, stabilizing generalization error as k grows.

Principles

• InfoNCE population risk approaches ideal cross-entropy.
• Increasing negative samples stabilizes generalization.
• Gâteaux differentiation quantifies generalization performance.

Method

The paper introduces a new continuity bound for InfoNCE loss via Gâteaux differentiation, preserving negative sample averaging and featuring an "inverse temperature" parameter to quantify generalization.

In practice

• Tune "inverse temperature" for algorithmic temperature.
• Increase k negative samples to stabilize generalization.

Topics

• Contrastive Learning
• InfoNCE Loss
• Similarity Search
• Generalization Error
• Gâteaux Differentiation

Best for: AI Scientist, Research Scientist

Related on AIssential

• Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation — Extensive-width shallow neural networks exhibit a phase transition between universal and specialization learning, impacting generalization and algorithmic tractability.
• Towards Accurate and Calibrated Classification: Regularizing Cross-Entropy From A Generative Perspective — Generative Cross-Entropy improves DNN accuracy and calibration by regularizing class-level confidence from a generative perspective.
• Mitigating Forgetting in Continual Learning with Selective Gradient Projection — SFAO uses similarity-gated gradient updates to balance plasticity and stability in continual learning, reducing forgetting and memory.
• InfoDensity: Rewarding Information-Dense Traces for Efficient Reasoning — High-quality LLM reasoning traces are informationally dense, showing low uncertainty convergence and monotonic progress in entropy reduction.
• A Stationary (and Therefore Compatible) Representation is All You Need — Stationary representations learned via d-Simplex fixed classifiers inherently ensure model compatibility over time.
• Fixing Unsupervised Hyperbolic Contrastive Loss [D] — Hyperbolic contrastive loss requires a higher temperature parameter than Euclidean loss due to different distance scales.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.