MoAKE: Toward Unified All-in-One Action Quality Assessment via Mixture of Action Knowledge Experts

2026-07-23 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, extended

Summary

MoAKE, a novel Mixture of Action Knowledge Experts framework, enables unified "all-in-one" Action Quality Assessment (AQA) for heterogeneous actions, overcoming the limitations of traditional "one-by-one" models that train separately for each action type. This approach mitigates negative knowledge transfer by employing specialized experts that learn complementary action patterns within a shared semantic space. Each expert utilizes segment-aware prototypes to manage varying temporal lengths and an Adaptive Intra- and Inter-Segment Relationship Modeling (AIISRM) module for multi-granularity temporal dynamics, dynamically aggregating knowledge for adaptive assessment. Benchmarks for all-in-one and zero/few-shot AQA demonstrate MoAKE's superior performance. It significantly outperforms existing methods on three long-term datasets (Rhythmic Gymnastics, Figure Skating, Artistic Swimming) and generalizes well on three short-term datasets (Diving, Skiing/Snowboarding, Surgery), achieving an average SRCC improvement of 1.2% and a 28.9% reduction in R-$\ell_{2}$ with 8.01M parameters and 1.44G FLOPs.

Key takeaway

For Machine Learning Engineers building Action Quality Assessment systems, if you need to handle diverse action types or generalize to new ones, MoAKE offers a unified, efficient solution. Its Mixture of Action Knowledge Experts architecture overcomes negative transfer, delivering superior performance and strong zero/few-shot generalization. You should consider MoAKE to develop more adaptive and scalable AQA systems for real-world applications.

Key insights

Unified AQA for diverse actions is achieved by adaptively mixing specialized expert knowledge to overcome negative transfer.

Principles

• Semantic differences hinder unified action quality assessment.
• Adaptive expert fusion enables positive knowledge transfer.
• Multi-granularity temporal modeling refines assessment.

Method

MoAKE employs a soft router to dynamically weight outputs from action knowledge experts. Each expert refines segment-aware prototypes via an AIISRM module for multi-granularity temporal modeling.

In practice

• Adopt Mixture-of-Experts for heterogeneous action assessment.
• Use segment-aware prototypes for variable video lengths.
• Model intra- and inter-segment relationships for detail.

Topics

• Action Quality Assessment
• Mixture of Experts
• Zero-shot Learning
• Few-shot Learning
• Video Understanding
• Temporal Modeling
• MoAKE Framework

Code references

• XuHuangbiao/MoAKE

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• MoAKE: Toward Unified All-in-One Action Quality Assessment via Mixture of Action Knowledge Experts — MoAKE unifies Action Quality Assessment for diverse actions using a mixture of specialized, dynamically aggregated experts.
• MoAKE: Toward Unified All-in-One Action Quality Assessment via Mixture of Action Knowledge Experts — MoAKE unifies Action Quality Assessment for diverse actions into a single model, overcoming "one-by-one" limitations.
• Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessment — DualAlign uses a two-stage fusion with adaptive alignment for robust multi-modal Action Quality Assessment.
• Divide, Deliberate, Decide: A Multi-Agent Framework for Fine-Grained Egocentric Action Recognition — Multi-agent VLM deliberation with heterogeneous specialists improves fine-grained egocentric action recognition by decorrelating model priors.
• HOI-aware Adaptive Network for Weakly-supervised Action Segmentation — Leveraging human-object interaction (HOI) context dynamically improves weakly-supervised action segmentation accuracy for ambiguous actions.
• PRISM: Synergizing Vision Foundation Models via Self-organized Expert Specialization — PRISM synergizes Vision Foundation Models through a dual-stream Mixture-of-Experts framework with self-organized expert specialization.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.