Fixing Unsupervised Hyperbolic Contrastive Loss [D]

2026-05-05 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, quick

Summary

A user implementing Unsupervised Hyperbolic Contrastive Loss on the ImageNet-1k dataset reported that their hyperbolic model achieved only 57% 1-NN accuracy, significantly underperforming a simple Euclidean contrastive loss which reached 64%. The user provided Python code for their `hb_contrastive_loss` function, which utilizes `expmap()` and `projx()` to ensure embeddings reside on the Lorentzian manifold, and specified a temperature parameter of 0.07, a batch size of 2048, and a learning rate of 1e-4. The core issue appears to be the low temperature parameter, which is unsuitable for the larger distance scales inherent in hyperbolic space.

Key takeaway

For machine learning engineers working with hyperbolic contrastive loss, ensure your temperature parameter is significantly higher than what you would use for Euclidean space, potentially in the 0.5 to 1.0 range, to account for the larger distance scales. Additionally, verify that manifold projection steps are applied not just during the forward pass but also after every gradient update to maintain embedding validity. Experiment with learning rate and batch size sweeps to further optimize performance.

Key insights

Hyperbolic contrastive loss requires a higher temperature parameter than Euclidean loss due to different distance scales.

Principles

• Hyperbolic distances are generally larger than Euclidean distances.
• Projection to the manifold is crucial after every gradient update.

In practice

• Increase temperature parameter for hyperbolic loss (e.g., 0.5-1.0).
• Ensure manifold projection after each gradient update.
• Sweep learning rate and batch size for optimization.

Topics

• Unsupervised Hyperbolic Contrastive Loss
• ImageNet-1k
• Lorentzian Manifold
• Contrastive Learning
• Hyperparameter Tuning

Best for: Machine Learning Engineer, AI Engineer

Related on AIssential

• Neural Collapse by Design: Learning Class Prototypes on the Hypersphere — Supervised learning can achieve optimal Neural Collapse geometry by unifying cross-entropy and contrastive methods on the hypersphere.
• Rapid mixing for Gibbs measures in Riemannian manifolds — Conditions for rapid mixing of Langevin dynamics on Riemannian manifolds are identified, achieving polynomial mixing times.
• Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate — Maximal Update (\$μ\$P) improves LLM hyperparameter transfer by maximizing the embedding layer's learning rate, a bottleneck in standard parameterization.
• Striding Across Reynolds Numbers: Representation Geometry in Neural PDE Generalisation — Representation geometry and multi-scale representations are critical for cross-Reynolds generalisation in neural PDE solvers.
• Neural Networks on Symmetric Spaces of Noncompact Type — A unified point-to-hyperplane distance enables novel neural network architectures on noncompact symmetric spaces.
• On-Manifold Variational Learning with Heat-Kernel Priors — Manifold-anchored variational learning improves medical image representation by keeping prototypes on the data manifold.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.