When AI watches and listens: Introducing SONIC-O1 for real-world audio-video understanding

2026-06-23 · Source: Vector Institute for Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, short

Summary

SONIC-O1 is an open, human-verified benchmark for evaluating multimodal large language models (MLLMs) on real-world audio-video understanding, addressing a gap in current evaluations focused on static images or short clips. It contains approximately 60 hours of content from 231 human-reviewed videos across 13 conversational topics and five domains, such as professional interactions and patient-doctor consultations. The benchmark assesses three capabilities: video summarization, evidence-grounded multiple-choice questions, and temporal localization with reasoning. Initial evaluations of leading closed-source and open-source MLLMs showed progress but also clear limitations. Closed-source models, particularly Gemini 3.0 Pro, performed best, achieving 25.4% R@0.5 on temporal localization, compared to 2.8% for Qwen3-Omni. Performance varied across domains and demographic groups, with a 21.4% gap for Gemini 3.0 Pro in temporal localization between Indigenous and Black participants, indicating uneven reliability.

Key takeaway

For Machine Learning Engineers developing or deploying multimodal LLMs in high-stakes applications, you must move beyond static image or short-clip evaluations. Use benchmarks like SONIC-O1 to rigorously test your models on long-form, real-world audio-video interactions, specifically focusing on temporal localization and group-wise performance disparities. This ensures your systems are fair and reliable across diverse demographics and complex conversational contexts, mitigating risks in areas like customer service or medical consultations.

Key insights

SONIC-O1 provides a human-verified benchmark for evaluating multimodal LLMs on real-world audio-video understanding, revealing current limitations.

Principles

• Multimodal AI needs real-world, long-form audio-video evaluation.
• Performance disparities exist across domains and demographics.
• Temporal localization remains a significant challenge for MLLMs.

Method

SONIC-O1 uses 231 human-reviewed videos (~60 hours) across 13 topics and 5 domains to evaluate MLLMs on video summarization, evidence-grounded multiple-choice questions, and temporal localization with reasoning.

In practice

• Test MLLMs on long, natural audio-video interactions.
• Conduct group-wise analysis for demographic fairness.
• Prioritize temporal localization improvements in MLLM development.

Topics

• Multimodal LLMs
• Audio-Video Understanding
• Benchmark Datasets
• Temporal Localization
• AI Fairness
• Gemini 3.0 Pro

Code references

• VectorInstitute/sonic-o1

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, AI Ethicist

Related on AIssential

• DEAF: A Benchmark for Diagnostic Evaluation of Acoustic Faithfulness in Audio Language Models — Audio MLLMs exhibit pervasive text dominance, prioritizing textual cues over acoustic signals, even when perceiving acoustic variations.
• Video = World + Event Stream — Wan-Streamer v0.3 models video as a persistent "world" and dynamic "event stream" for real-time predictive interaction.
• OmniReasoner: Thinking with Long Audio-Video via Native Tool Use — OmniReasoner enables omnimodal LLMs to reason with long audio-video by dynamically zooming into relevant temporal segments.
• MMAE: A Massive Multitask Audio Editing Benchmark — The audio editing evaluation landscape is fragmented, and current models perform poorly on a new comprehensive benchmark.
• M$^3$Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions — M$^3$Exam benchmarks multimodal memory for realistic user-agent interactions, revealing key gaps and proposing M$^3$Proctor for efficiency.
• Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos — AV-Flamingo is an open AV-LLM excelling at long, complex video understanding through novel data, training, and reasoning frameworks.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Vector Institute for Artificial Intelligence.