Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessment

2026-07-08 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

DualAlign, a two-stage multi-modal fusion framework with adaptive alignment, addresses challenges in Action Quality Assessment (AQA) by improving cross-modal alignment and fusion stability. It first creates a coherent visual representation from RGB video, optical flow, and skeleton modalities, then integrates textual semantics to complement the visual data without distortion. To facilitate evaluation, the framework introduces MM--JDM, a new dataset featuring modality noise, class imbalance, and label scarcity. Experiments demonstrate DualAlign's effectiveness, showing a 21.16% average correlation improvement on MM--JDM over existing methods, alongside gains of 3.53% on RG and 5.95% on Fis-V benchmarks. The framework also maintains robustness under conditions of missing modalities and limited labels.

Key takeaway

For Machine Learning Engineers developing Action Quality Assessment systems, DualAlign demonstrates that a two-stage multi-modal fusion with adaptive alignment significantly improves performance. You should prioritize establishing coherent visual representations before integrating textual semantics. This approach, robust to missing modalities and label scarcity, offers substantial gains, improving correlation by 21.16% on challenging datasets like MM--JDM.

Key insights

DualAlign uses a two-stage fusion with adaptive alignment for robust multi-modal Action Quality Assessment.

Principles

• Maximize shared structural information across visual modalities.
• Incorporate textual semantics after visual stabilization.
• Robustness under missing modalities and label scarcity is crucial.

Method

DualAlign first constructs a coherent visual representation from RGB, optical flow, and skeleton, then incorporates textual semantics after visual stabilization to prevent distortion.

In practice

• Utilize two-stage fusion for heterogeneous multi-modal data.
• Prioritize visual coherence before adding high-level text.
• Design systems robust to missing modalities and scarce labels.

Topics

• Action Quality Assessment
• Multi-Modal Fusion
• Adaptive Alignment
• RGB Video
• Optical Flow
• Skeleton Tracking
• MM--JDM Dataset

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• A Multi-Modal Framework with Cross-Subject Pseudo-Labeling and Semantic Alignment for Micro-Gesture Recognition — A multi-modal framework uses pseudo-labeling and semantic alignment to improve micro-gesture recognition across subjects.
• Improving Viewpoint-Invariance and Temporal Consistency for Action Detection — A two-stage approach using augmented virtual viewpoints and a multi-scale temporal encoder improves action detection's viewpoint-invariance and temporal consistency.
• DAMAGESCOPE: How We Built a Dual-Encoder Damage Detection Model That Works Through Clouds — Adaptive cross-modal attention fusion enables robust damage detection from satellite imagery despite cloud occlusion.
• TIGER: Taming Identity, Geometry, and Generative Priors for High-Quality Face Video Restoration — TIGER fuses identity, geometry, and generative priors to restore high-fidelity, identity-consistent face videos efficiently.
• See More, Match Better: Multi-Source Feature Fusion for Two-View Correspondence Learning — TriMatch fuses geometric, texture, and structural semantic features to overcome pseudo-consistent outliers in two-view correspondence.
• Divide, Deliberate, Decide: A Multi-Agent Framework for Fine-Grained Egocentric Action Recognition — Multi-agent VLM deliberation with heterogeneous specialists improves fine-grained egocentric action recognition by decorrelating model priors.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.