Conan-embedding-v3: Fusing Modality-Specific Models for Omni-Modal Embedding

2026-06-08 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Conan-embedding-v3 is a novel decouple--fuse--recover framework designed for omni-modal retrieval, aiming to create a single embedding space for text, image, video, document, and audio inputs. The system first employs "Decoupled Specialist Fusion," independently training modality specialists and then fusing their task vectors into a unified dense backbone. While this strategy effectively combines visual, video, and document retrieval capabilities, it introduces a "Projector Drift" failure for projector-based modalities like audio, causing significant performance regression despite preserving audio-specific modules. To address this, Conan-embedding-v3 implements "Projector Recovery," involving full-parameter fine-tuning of the projector while keeping the backbone frozen, followed by balanced multi-modal rehearsal. The resulting model successfully integrates these diverse retrieval pathways into one backbone, achieving 74.9 scores on MMEB and 55.61 on the 30-task MAEB audio suite.

Key takeaway

For Machine Learning Engineers building omni-modal retrieval systems, you should anticipate "Projector Drift" when fusing modality-specific backbones, particularly for projector-based inputs like audio. This phenomenon can cause significant performance regression even if modality-specific modules remain unchanged. To mitigate this, consider implementing a "Projector Recovery" phase, fine-tuning only the projector while keeping the fused backbone frozen, followed by balanced multi-modal rehearsal to ensure robust performance across all integrated modalities.

Key insights

Fusing modality-specific models for omni-modal embeddings requires addressing "Projector Drift" through targeted recovery.

Principles

• Decoupled Specialist Fusion composes modality capabilities.
• Projector Drift causes regression in fused projector-based systems.

Method

Conan-embedding-v3 uses a decouple--fuse--recover framework: train specialists, fuse task vectors into a backbone, then apply full-parameter fine-tuning of projectors with a frozen backbone, followed by multi-modal rehearsal.

Topics

• Omni-modal Embedding
• Multi-modal Retrieval
• Modality Fusion
• Projector Drift
• Conan-embedding-v3
• Deep Learning Architectures

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Conan-embedding-v3: Fusing Modality-Specific Models for Omni-Modal Embedding — Conan-embedding-v3 fuses modality specialists for omni-modal embeddings, but requires "Projector Recovery" to prevent "Projector Drift" in projector-based modalities.
• Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMs — Static fusion in Omni-MLLMs causes performance paradoxes, necessitating dynamic orchestration for robust multimodal integration.
• To Be Multimodal or Not to Be: Query-Adaptive Audio-Visual Person Retrieval via Active Modality Detection — Cross-modal score consistency reliably detects active modalities, preventing noise from absent modalities in multimodal retrieval.
• MASER: Modality-Adaptive Specialist Routing for Embodied 3D Spatial Intelligence — Dynamically selecting the best modality adapter based on question semantics improves 3D spatial reasoning for embodied agents.
• Context Unrolling in Omni Models — Unified multimodal training enables "Context Unrolling" for enhanced cross-modal reasoning and performance.
• Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction — Staged multimodal fusion with independent encoder pre-training improves robustness for continuous emotion intensity prediction.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.