$C^3$ASD: Multi-Level Consistency-Driven Representation Learning

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Multimodal AI · Depth: Expert, extended

Summary

Chung-Ang University researchers developed $C^3$ASD, a multi-level consistency-driven framework for robust Active Speaker Detection (ASD). This framework addresses the limitation of existing audio-visual fusion methods that degrade under real-world corruptions like background noise, occlusion, or simultaneous modality degradation due to a lack of explicit consistency constraints. $C^3$ASD introduces three complementary constraints: embedding-level inter-modality consistency, sequence-level intra-modality consistency via track-aware contrastive learning, and prediction-level consistency through knowledge distillation. Experiments show $C^3$ASD achieves 93.8% mAP on AVA-ActiveSpeaker with only 1.02M parameters and 0.62G FLOPs. It also outperforms baselines on the WASD dataset with 86.1% mAP and demonstrates significant robustness gains under diverse audio, visual, and joint corruptions, including +2.04% mAP under object occlusion and +1.23% average mAP with MUSAN noise and occlusion.

Key takeaway

Machine Learning Engineers developing robust audio-visual systems and facing real-world data corruptions or cross-domain generalization challenges should consider integrating multi-level consistency regularization into their models. This approach enhances robustness against diverse audio, visual, and joint corruptions, improving performance on noisy data and in-the-wild scenarios without needing corrupted training samples or complex architectural changes. Prioritize speaking-aware cross-modal alignment and track-aware intra-modal clustering for optimal results.

Key insights

Robust Active Speaker Detection requires multi-level consistency constraints to align cross-modal and structure intra-modal representations against real-world corruptions.

Principles

• Multimodal robustness benefits from explicit consistency constraints.
• Cross-modal alignment should be speaking-aware.
• Intra-modal clusters need track-aware separation.

Method

$C^3$ASD applies embedding-level inter-modality cosine similarity, sequence-level track-aware supervised contrastive learning, and prediction-level knowledge distillation via MSE with confidence masking to audio-visual streams.

In practice

• Integrate consistency losses into existing ASD architectures.
• Apply track-aware contrastive learning for multi-speaker data.
• Use confidence masking for robust knowledge distillation.

Topics

• Active Speaker Detection
• Multimodal Learning
• Representation Learning
• Consistency Regularization
• Audio-Visual Fusion
• Robustness
• Knowledge Distillation

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Joint-Centric Dual Contrastive Alignment with Structure-Preserving and Information-Balanced Regularization — HILBERT aligns audio-text representations for long sequences by balancing modality contributions and preserving structure.
• Speaker-Invariant Representation Learning for Spoofing Detection via Gradient Reversal and A Variational Information Bottleneck — A teacher-student framework with gradient reversal and a Variational Information Bottleneck improves spoofing detection by learning speaker-invariant representations.
• Sublinearly Structured Deep Neural Networks Achieve Feature Learning Consistency for Compositional Functions — Sublinearly structured DNNs achieve feature-learning consistency and universal approximation for compositional functions, even when over-parameterized.
• To Be Multimodal or Not to Be: Query-Adaptive Audio-Visual Person Retrieval via Active Modality Detection — Query-adaptive modality detection significantly improves person retrieval by preventing noise from absent modalities.
• Domain Generalizable Adaptation of 3D Vision-Language Models via Regularized Fine-Tuning — ReFine3D regularizes 3D LMM fine-tuning with multi-view consistency and text diversity to prevent overfitting and enhance domain generalization.
• MIC: Maximizing Informational Capacity in Adaptive Representations via Isotropic Subspace Alignment — MIC enhances multi-scale representations by aligning isotropic subspaces and mitigating redundancy through regularization and self-distillation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.