MoAKE: Toward Unified All-in-One Action Quality Assessment via Mixture of Action Knowledge Experts

2026-07-22 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

MoAKE, a novel Mixture of Action Knowledge Experts framework, addresses the limitations of traditional "one-by-one" Action Quality Assessment (AQA) methods by enabling unified, all-in-one evaluation of heterogeneous actions within a single model. Existing AQA approaches require separate models for each action type, limiting real-world deployment and generalization. MoAKE mitigates negative knowledge transfer by learning complementary experts that capture diverse action patterns in a shared semantic space, dynamically aggregating their knowledge for input-specific assessment. Each expert incorporates segment-aware prototypes to handle varying temporal lengths and an Adaptive Intra- and Inter-Segment Relationship Modeling (AIISRM) module for multi-granularity temporal dynamics. Extensive experiments demonstrate MoAKE's significant outperformance on three long-term datasets in the all-in-one setting, alongside consistent generalization on three short-term datasets under zero/few-shot evaluation.

Key takeaway

For Computer Vision Engineers building robust video analysis systems for diverse actions, MoAKE presents a compelling unified model approach. Its Mixture of Action Knowledge Experts architecture mitigates negative knowledge transfer, improving generalization across heterogeneous actions compared to "one-by-one" models. You should explore MoAKE's design, particularly its dynamic expert aggregation and AIISRM module, to simplify deployment and enhance performance in all-in-one or zero/few-shot Action Quality Assessment scenarios.

Key insights

MoAKE unifies Action Quality Assessment for diverse actions using a mixture of specialized, dynamically aggregated experts.

Principles

• Mitigate negative knowledge transfer across diverse tasks.
• Dynamically aggregate expert knowledge for adaptive assessment.
• Model multi-granularity temporal dynamics effectively.

Method

MoAKE employs a Mixture of Action Knowledge Experts, each with segment-aware prototypes and an Adaptive Intra- and Inter-Segment Relationship Modeling (AIISRM) module, to capture diverse action patterns and dynamically aggregate knowledge for unified AQA.

In practice

• Develop unified models for heterogeneous video tasks.
• Apply expert aggregation for adaptive system responses.
• Benchmark all-in-one and zero/few-shot AQA.

Topics

• Action Quality Assessment
• Mixture-of-Experts
• Computer Vision
• Video Understanding
• Zero-shot Learning
• Temporal Modeling

Code references

• XuHuangbiao/MoAKE

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• MoAKE: Toward Unified All-in-One Action Quality Assessment via Mixture of Action Knowledge Experts — Unified AQA for diverse actions is achieved by adaptively mixing specialized expert knowledge to overcome negative transfer.
• Divide, Deliberate, Decide: A Multi-Agent Framework for Fine-Grained Egocentric Action Recognition — Multi-agent VLM deliberation with heterogeneous specialists improves fine-grained egocentric action recognition by decorrelating model priors.
• PRISM: Synergizing Vision Foundation Models via Self-organized Expert Specialization — PRISM synergizes Vision Foundation Models through a dual-stream Mixture-of-Experts framework with self-organized expert specialization.
• Unified Video Dense Prediction from Disjoint Data — UniD unifies eight video dense prediction tasks using disjoint data via diffusion model priors and expert distillation.
• Understanding and Mitigating the Video-Action Generalization Gap via Temporal Ratio — The video-action generalization gap stems from temporal attention shifts, addressable via adaptive guidance.
• MoAKE: Toward Unified All-in-One Action Quality Assessment via Mixture of Action Knowledge Experts — MoAKE unifies Action Quality Assessment for diverse actions into a single model, overcoming "one-by-one" limitations.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.