Contrastive-Collapsed Loss for Flexible and Geometrically Optimal Embeddings and Faster Convergence

2026-07-14 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

CoCo Loss is a novel loss function designed to learn normalized and well-structured representations by encouraging intra-class collapse and inter-class contrast. It provides neural networks with the flexibility to approximate geometrically optimal embeddings, ensuring large angular separation between classes. A theoretical analysis positions CoCo Loss favorably against objectives like dot regression and cross-entropy, demonstrating benefits such as closer initialization to optimal configurations, more informative gradients, and stronger incentives for class-wise representation collapse. Extensive experiments on diverse tabular datasets from the OpenML-CC18 benchmark show CoCo Loss achieves competitive performance compared to leading methods, including kernel SVM, Random Forest, dot regression, and cross-entropy-based neural networks, while also promoting tighter class clustering and faster convergence.

Key takeaway

For Machine Learning Engineers developing classification models, especially with tabular data, CoCo Loss offers a compelling alternative to standard objectives. You should consider integrating CoCo Loss into your neural network architectures. This can yield tighter class clustering, more discriminative representations, and faster training convergence. Evaluate its performance against cross-entropy or dot regression in your specific applications to optimize embedding quality and training efficiency.

Key insights

CoCo Loss optimizes embeddings for better class separation and faster training.

Principles

• Intra-class collapse improves representation structure.
• Inter-class contrast enhances class separation.
• Optimal initialization accelerates convergence.

In practice

• Implement CoCo Loss in neural network training.
• Test CoCo Loss on tabular classification tasks.

Topics

• CoCo Loss
• Loss Functions
• Embeddings
• Neural Networks
• Tabular Data
• Classification

Best for: AI Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Neural Collapse by Design: Learning Class Prototypes on the Hypersphere — Supervised learning can achieve optimal Neural Collapse geometry by unifying cross-entropy and contrastive methods on the hypersphere.
• CoCo-Fed: A Unified Framework for Memory- and Communication-Efficient Federated Learning at the Wireless Edge — CoCo-Fed unifies local memory efficiency and global communication reduction in federated learning for wireless edge networks.
• Experimental Analysis of Neural Network-Based Image Classification on the CIFAR-10 Dataset — Neural network image classification on CIFAR-10 reveals the practical distinction between representation learning and memorization.
• [P] Graph Representation Learning Help — Representation collapse in JEPA-style models often stems from predictor capacity or insufficient objective diversity.
• Loss Landscape Topology Reveals Why Simple Baselines are Competitive at 3D Point Cloud Segmentation Under Class Imbalance — Loss landscape topology, shaped by class imbalance, limits the effectiveness of specialized methods in 3D point cloud segmentation.
• Similarity search generalisation in contrastive learning with InfoNCE loss — InfoNCE loss's population risk is O(1/k) close to an expected cross-entropy, stabilizing generalization error as k grows.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.