CANDOR: Chance-Calibrated Discordance in Frozen Foundation Encoders

2026-07-20 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

CANDOR, a novel chance-calibrated discordance measure, evaluates frozen foundation encoders by addressing biases in existing nearest-neighbor methods. Traditional approaches misinterpret encoder "blindness" due to unequal bank sizes, where label prevalence, not feature geometry, dictates outcomes. CANDOR rectifies this by employing equal-size banks, ensuring symmetry under label swaps and fixing its chance level at exactly one half. Applied across 22 encoders, 20 datasets from 7 domains, and 605,443 images, this correction reveals that "collapse" falls below chance almost universally, indicating encoders are weak but not blind. For instance, a chest model achieving 84.5 AUROC for pneumothorax still places 18.4% of positives nearer opposite-label films. The same encoder resolves bird species at 4.5 but struggles with chest findings at 42.8 and glaucoma at 49.8, suggesting a "deficit is selection, not information." CANDOR's fixed chance level allows it to flag poorly supported findings before any head is trained.

Key takeaway

For Machine Learning Engineers evaluating frozen foundation encoders, CANDOR offers a critical diagnostic tool. You should integrate CANDOR into your pre-training workflow to accurately assess an encoder's inherent feature separation for specific tasks. This allows you to identify findings poorly supported by the encoder before investing in head training, potentially guiding better model selection or indicating where custom heads are most needed to extract existing information.

Key insights

CANDOR accurately assesses frozen encoder feature separation by correcting for label prevalence bias.

Principles

• Encoder evaluation needs geometry, not just head performance.
• Unequal label banks bias discordance measures.
• Fixed chance levels enable pre-training assessment.

Method

CANDOR measures discordance using equal-size, symmetric banks to fix the chance level at 0.5, revealing true feature separation independent of label prevalence.

In practice

• Use CANDOR to pre-screen encoder suitability for tasks.
• Identify findings poorly supported by a frozen encoder.
• Guide head selection for specific encoder features.

Topics

• CANDOR
• Foundation Models
• Encoder Evaluation
• Feature Geometry
• Discordance Measures
• Machine Learning Diagnostics

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Rethinking Noise-Robust Training for Frozen Vision Foundation Models: A Cross-Dataset Benchmark with a Case Study of Small-Loss Failure — No universal noisy-label learning method exists for frozen Vision Foundation Models; selection must be regime-aware.
• Evaluation-Strategy Gap in Fault Diagnosis of Deep Learning Programs — Fault diagnosis in DL programs exhibits an evaluation gap when applied to unseen code, driven by feature structure.
• Expert-Guided Class-Conditional Goodness-of-Fit Scores for Interpretable Classification with Informative Missingness: An Application to Seismic Monitoring — Expert-guided, interpretable features derived from class-conditional models improve classification, especially with informative missingness.
• Look Again Before You Abstain:Budgeted Conformal Evidence Acquisition for Reliable Vision-Language Model — BCEA improves LVLM reliability by acquiring more visual evidence under budget, restoring statistical guarantees through recalibration.
• Noise-Aware Framework for Correcting Corrupted Labels — CANOLA corrects corrupted labels by estimating noise and iteratively refining them for robust model training.
• Detecting Explanatory Insufficiency in Learned Representations: A Framework for Representational Vigilance — VER identifies persistent residual structures in learned representations to detect explanatory insufficiency beyond standard metrics.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.