Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio

2026-07-21 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

The Fusion Embedding family introduces a unified embedding space for text, images, video, and audio, allowing a single index to serve diverse user queries. This system integrates audio into a frozen vision-language embedding base, which already handles text, images, and video, without updating the base parameters. Fusion-embedding-1 employs a 16.4M-parameter connector between a frozen audio tower and the base. Fusion-embedding-2 enhances this with 44.2M-parameter modality-gated deep adapters that do not execute on text, image, or video inputs, ensuring bit-for-bit output consistency with the original base. Critically, aligning audio to text alone enables emergent audio-image retrieval without requiring paired audio-visual training data. The authors also detail design space exploration, including controlled negative results showing that LLM-rewritten captions, alternative audio towers, or wider connectors reduce retrieval performance. Both generations train in hours on a single GPU, with weights, code, and evaluation harness openly released.

Key takeaway

For Machine Learning Engineers developing multimodal retrieval systems, Fusion Embedding offers a compelling strategy to integrate audio efficiently. You can extend existing vision-language models to include audio by training minimal connector parameters, avoiding costly full model retraining. This approach enables emergent audio-image retrieval without needing paired audio-visual data, significantly reducing data acquisition burdens. Consider adopting this method to rapidly expand your system's modality support and enhance user query capabilities.

Key insights

Aligning audio to a frozen vision-language base via text enables emergent cross-modal retrieval without paired audio-visual data.

Principles

• Freezing a vision-language base preserves existing capabilities.
• Aligning one new modality to text can unlock others.
• Simpler connectors can outperform complex ones.

Method

Fusion Embedding connects a frozen audio tower to a frozen vision-language embedding base using a small connector (16.4M params) or modality-gated deep adapters (44.2M params), aligning audio to text.

In practice

• Integrate audio into existing multimodal retrieval systems.
• Explore small, specialized connectors for new modalities.
• Leverage text alignment for zero-shot cross-modal capabilities.

Topics

• Fusion Embedding
• Multimodal Embeddings
• Cross-Modal Retrieval
• Audio-Text Alignment
• Vision-Language Models
• Parameter-Efficient Training

Best for: Research Scientist, AI Architect, AI Product Manager, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Vortex: Multi-Modal Fusion System for Intelligent Video Retrieval — Vortex fuses CLIP and SigLIP2 embeddings with relevance feedback for effective multimodal video retrieval.
• To Be Multimodal or Not to Be: Query-Adaptive Audio-Visual Person Retrieval via Active Modality Detection — Cross-modal score consistency reliably detects active modalities, preventing noise from absent modalities in multimodal retrieval.
• CMDR: Contextual Multimodal Document Retrieval — Contextual multimodal document retrieval requires modeling cross-page dependencies for accurate information retrieval.
• Multimodal Embedding & Reranker Models with Sentence Transformers — Sentence Transformers v5.4 introduces multimodal embedding and reranking for text, images, audio, and video.
• Embed the world: Multimodal AI for searchable aerial imagery at scale — Multimodal AI, combining image embeddings and LLM captions, enables efficient natural-language search over complex aerial imagery.
• Conan-embedding-v3: Fusing Modality-Specific Models for Omni-Modal Embedding — Conan-embedding-v3 fuses modality specialists for omni-modal embeddings, but requires "Projector Recovery" to prevent "Projector Drift" in projector-based modalities.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.