HumanOmni-Speaker: Identifying Who said What and When

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

HumanOmni-Speaker is a new unified Omni-modal Large Language Model from Alibaba Group and Shenzhen University of Advanced Technology, designed to accurately identify "Who said what and when" in complex multi-person conversations. It addresses the "illusion of competence" in current models that exploit visual biases and sparse visual sampling (1-2 fps). The model introduces a Visual Delta Encoder, which samples raw video at 25 fps and compresses inter-frame motion residuals into just 6 tokens per frame, capturing fine-grained visemes and speaker trajectories. Alongside this, the Visual-Registered Speaker Diarization and Recognition (VR-SDR) benchmark is proposed to eliminate visual shortcuts and demand true end-to-end spatio-temporal identity binding. HumanOmni-Speaker demonstrates superior performance, enabling end-to-end lip-reading and high-precision spatial localization without intrusive cropping, outperforming open-source models and competing with Gemini3-Pro.

Key takeaway

For Machine Learning Engineers developing multimodal LLMs for human-centric interaction, you should prioritize architectural designs that capture high-frequency visual dynamics. Relying on sparse visual sampling leads to an "illusion of competence"; instead, consider delta encoding techniques like the Visual Delta Encoder to achieve true spatio-temporal alignment. Your evaluation should incorporate rigorous benchmarks such as VR-SDR, which eliminate visual shortcuts and demand genuine cross-modal understanding for tasks like speaker diarization and lip-reading.

Key insights

Omni-modal LLMs require high-frequency visual dynamics and rigorous evaluation to accurately decipher complex multi-person conversations.

Principles

• Sparse visual sampling (1-2 fps) creates an "illusion of competence" in multimodal LLMs.
• Benchmarks must eliminate visual shortcuts to force genuine cross-modal understanding.
• Explicitly encoding inter-frame motion residuals is key for fine-grained spatio-temporal alignment.

Method

HumanOmni-Speaker employs a Visual Delta Encoder that samples raw video at 25 fps, compressing inter-frame motion into 6 tokens per frame, then aligns these with an LLM via a three-stage training pipeline.

In practice

• Enables end-to-end lip-reading from raw video.
• Achieves high-precision speaker localization.
• Supports robust speaker diarization and recognition.

Topics

• Omni-modal LLMs
• Speaker Diarization
• Visual Delta Encoder
• Lip-reading
• Speaker Localization
• Multimodal Benchmarking

Code references

• m-bain/whisperX

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents — Converting text-based tool-calling benchmarks to audio reveals significant model- and task-dependent performance gaps.
• The Lipreading Gap: Do VSR Models Perceive Visual Speech Like Human Lipreaders? — VSR models outperform humans by exploiting linguistic patterns, not human-like visual speech perception.
• LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning — Unified latent space reasoning preserves dense audio-visual information, outperforming text-centric Chain-of-Thought.
• LoMo: Local Modality Substitution for Deeper Vision-Language Fusion — LoMo addresses VLM "carrier sensitivity" by creating semantically equivalent text and image representations through local modality substitution.
• Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs — Omnimodal LLMs struggle to reject textual claims contradicting their sensory input, despite internal encoding of mismatches.
• LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokens — LLaMo unifies motion-language understanding and generation in LLMs using continuous tokens and a MoT architecture.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.