OmniGCD: Abstracting Generalized Category Discovery for Modality Agnosticism

2026-04-16 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

OmniGCD introduces a novel modality-agnostic approach to Generalized Category Discovery (GCD), a task involving identifying both known and novel classes from partially labeled data. Unlike previous GCD methods that are modality-specific and require fine-tuning per dataset, OmniGCD uses modality-specific encoders (e.g., vision, audio, text, remote sensing) to create a shared GCD latent space. This latent space is then transformed at test-time by a synthetically trained Transformer-based model to enhance clustering. The researchers evaluated OmniGCD in a zero-shot GCD setting, where no dataset-specific fine-tuning is permitted. Trained once on synthetic data, OmniGCD achieved an average classification accuracy improvement of +6.2, +17.9, +1.5, and +12.7 percentage points for vision, text, audio, and remote sensing modalities, respectively, across 16 datasets.

Key takeaway

For research scientists developing machine learning models for diverse data types, OmniGCD offers a path to more scalable and generalized category discovery. By leveraging modality-agnostic methods, you can develop encoders independently of the GCD task, significantly reducing the need for dataset-specific fine-tuning and accelerating model deployment across various applications. Consider integrating OmniGCD's approach to improve classification accuracy for both known and novel classes in zero-shot scenarios.

Key insights

OmniGCD enables zero-shot, modality-agnostic category discovery by decoupling representation learning from category identification.

Principles

• Strong encoders are crucial for modality-agnostic GCD.
• Synthetic data training can enable zero-shot generalization.

Method

OmniGCD processes inputs via modality-specific encoders, reduces dimensions to a GCD latent space, and transforms this space using a synthetically trained Transformer for clustering.

In practice

• Apply OmniGCD for cross-modal category discovery.
• Utilize strong pre-trained encoders for diverse data types.

Topics

• Generalized Category Discovery
• Modality-Agnostic Learning
• OmniGCD
• Zero-Shot Learning
• Latent Space

Code references

• Jordan-HS/OmniGCD

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• OmniGCD: Abstracting Generalized Category Discovery for Modality Agnosticism — OmniGCD enables zero-shot, modality-agnostic category discovery by decoupling representation learning from category discovery.
• Nothing from Something: Can a Language Model Discover 0? — Language models struggle to discover zero independently but improve with few-shot training, especially with pretraining.
• The "Final Boss" of Deep Learning — Intrinsic architectural improvements, not just tool use, are crucial for robust algorithmic reasoning in language models.
• BRo-JEPA: Learning Modular Arithmetic in Latent Space — Latent world models can learn symbolic rules if their architecture aligns with the problem's inherent structure.
• Learning Emergent Modular Representations in Multi-modality Medical Vision Foundation Models — Regulating specialization and coordination in modular networks overcomes Non-IID challenges in multi-modality medical vision foundation models.
• Weasel: Out-of-Domain Generalization for Web Agents via Importance-Diversity Data Selection — Weasel enhances web agent generalization and efficiency through importance-diversity trajectory selection and state pruning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.