MIBURI: Towards Expressive Interactive Gesture Synthesis

2026-03-03 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Human-Computer Interaction · Depth: Expert, quick

Summary

MIBURI is introduced as the first online, causal framework designed for generating expressive full-body gestures and facial expressions synchronized with real-time spoken dialogue. This system addresses limitations in current Embodied Conversational Agents (ECAs), which often produce rigid or low-diversity motions, and generative co-speech gesture synthesis methods that rely on future speech context and long run-times. MIBURI utilizes body-part aware gesture codecs to encode hierarchical motion details into multi-level discrete tokens. These tokens are then autoregressively generated by a two-dimensional causal framework, conditioned on LLM-based speech-text embeddings, to model both temporal dynamics and part-level motion hierarchy in real time. The framework incorporates auxiliary objectives to promote expressive and diverse gestures while avoiding static poses, with comparative evaluations demonstrating its ability to produce natural and contextually aligned gestures.

Key takeaway

For research scientists developing Embodied Conversational Agents, MIBURI offers a novel approach to overcome limitations in gesture expressiveness and real-time synchronization. You should explore integrating causal, body-part aware gesture codecs and LLM-based speech-text embeddings to achieve more natural and diverse human-like interactions, moving beyond rigid or context-dependent motion generation.

Key insights

MIBURI is a causal, real-time framework for generating expressive, synchronized full-body gestures and facial expressions for conversational agents.

Principles

• Causal generation enables real-time interaction.
• Hierarchical motion encoding improves expressiveness.

Method

MIBURI employs body-part aware gesture codecs to create multi-level discrete tokens, which are then autoregressively generated by a 2D causal framework conditioned on LLM-based speech-text embeddings.

In practice

• Integrate LLM-based speech-text embeddings.
• Utilize auxiliary objectives for gesture diversity.

Topics

• Embodied Conversational Agents
• Gesture Synthesis
• Real-time AI
• Causal AI Frameworks
• LLM Embeddings

Best for: Research Scientist, AI Researcher, AI Scientist, AI Engineer

Related on AIssential

• LLM-Powered Interactive Robotic Action Synthesis from Multimodal Speech, Gestures, and Music — LLMs can synthesize complex robot actions from multimodal human inputs like speech, gestures, and music.
• PhysDrift: Bridging the Embodiment Gap in Humanoid Co-Speech Motion Generation — Directly generating robot-native co-speech motions from speech overcomes the embodiment gap, improving humanoid expressiveness.
• Embodied Conversational Systems in Human–Robot Interaction: Introduction to the Special Issue — Embodied Conversational Systems enhance Human-Robot Interaction through multimodal communication and social intelligence.
• Plan, Don't Pose: Long Composite Motion Generation with Text-Aligned BFM — Text2BFM decouples semantic planning from motion execution for robust, efficient text-to-motion generation.
• DiscoForcing: A Unified Framework for Real-Time Audio-Driven Character Control with Diffusion Forcing — DiscoForcing enables real-time, stable audio-driven character animation by combining causal encoding with diffusion forcing and history-guided sampling.
• M3T: Discrete Multi-Modal Motion Tokens for Sign Language Production — Integrating high-dimensional facial models and collapse-free tokenization is crucial for linguistically complete sign language production.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.