AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation

2026-04-09 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, quick

Summary

AVGen-Bench is a new task-driven benchmark designed for evaluating Text-to-Audio-Video (T2AV) generation models, addressing the limitations of existing fragmented evaluation methods. It features high-quality prompts categorized into 11 real-world scenarios. The benchmark introduces a multi-granular evaluation framework that integrates specialist models with Multimodal Large Language Models (MLLMs) to assess T2AV generation from perceptual quality to fine-grained semantic control. Initial evaluations using AVGen-Bench reveal a significant disparity between the strong audio-visual aesthetics of current T2AV models and their weak semantic reliability. Specific weaknesses include consistent failures in text rendering, speech coherence, physical reasoning, and a universal inability to control musical pitch.

Key takeaway

For research scientists developing or deploying Text-to-Audio-Video (T2AV) generation models, you should integrate AVGen-Bench into your evaluation pipeline. This benchmark will help you identify specific weaknesses in semantic reliability, such as text rendering, speech coherence, and musical pitch control, guiding your model improvements beyond mere aesthetic quality.

Key insights

AVGen-Bench offers a multi-granular evaluation framework for T2AV models, revealing a gap between aesthetics and semantic control.

Principles

• T2AV evaluation needs multi-granular assessment.
• Combine specialist models with MLLMs for comprehensive T2AV evaluation.

Method

AVGen-Bench uses 11 real-world prompt categories and a multi-granular framework combining specialist models with MLLMs to evaluate T2AV generation for perceptual quality and semantic controllability.

In practice

• Use AVGen-Bench for T2AV model comparison.
• Focus T2AV development on semantic reliability.

Topics

• Text-to-Audio-Video Generation
• AVGen-Bench
• Multimodal Large Language Models
• Benchmark Evaluation
• Semantic Controllability

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Text2CAD-Bench: A Benchmark for LLM-based Text-to-Parametric CAD Generation — Text2CAD-Bench systematically evaluates LLM-based text-to-CAD generation across diverse geometric complexities and real-world applications.
• MVEB: Massive Video Embedding Benchmark — The MVEB benchmark reveals diverse video embedding model performance, with no single architecture dominating across all 23 tasks.
• MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generation — Cinematic expressiveness in multi-talker audio-video generation requires evaluation beyond basic audio-visual alignment.
• IndicContextEval: A Benchmark for Evaluating Context Utilisation in Audio Large Language Models Across 8 Indic Languages — AudioLLMs' actual context utilization from prompts is ambiguous, necessitating explicit evaluation benchmarks like IndicContextEval to reveal true grounding.
• MMAE: A Massive Multitask Audio Editing Benchmark — The audio editing evaluation landscape is fragmented, and current models perform poorly on a new comprehensive benchmark.
• WeGenBench: A Multidimensional Diagnostic Benchmark towards Text-to-Image Model Optimization — WeGenBench offers a multi-dimensional, VLM-integrated benchmark to precisely diagnose text-to-image model deficiencies across diverse linguistic and cultural contexts.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.