FacePlex: Full-Duplex Joint Speech-Facial Motion Generation for Conversational Avatars

2026-06-29 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, medium

Summary

FacePlex is a novel unified streaming framework designed for full-duplex joint speech-facial motion generation, addressing the challenge of real-time synchronized speech and facial animation for conversational avatars. Existing systems either generate speech without facial motion or animate faces from pre-existing audio. FacePlex formalizes this joint generation, producing speech tokens and facial motion tokens simultaneously at each step. It incorporates two key components: Rolling Flow Matching, which adapts flow matching for online motion generation by committing new motion frames at every streaming step, and Rolling Cross-Attention, which links the streaming audio and motion queues, enabling mutual conditioning as generation progresses. Experiments, ablation studies, and a user study confirm FacePlex's ability to achieve full-duplex joint speech-facial motion generation under online streaming constraints, demonstrating superior lip-sync quality and motion fidelity compared to audio-driven facial motion baselines.

Key takeaway

For AI Scientists and NLP Engineers developing real-time conversational avatar systems, FacePlex offers a critical advancement. You should consider integrating its full-duplex joint speech-facial motion generation approach to overcome limitations of existing partial solutions. This method significantly improves lip-sync quality and motion fidelity under online streaming constraints, enabling more natural and engaging virtual interactions. Evaluate its Rolling Flow Matching and Rolling Cross-Attention components for your next-generation avatar projects.

Key insights

FacePlex enables real-time, synchronized speech and facial motion generation for conversational avatars through a unified streaming framework.

Principles

• Jointly generate speech and facial motion tokens.
• Adapt flow matching for online motion generation.
• Mutually condition audio and motion streams.

Method

FacePlex uses Rolling Flow Matching to commit new motion frames online and Rolling Cross-Attention to couple audio and motion queues, allowing real-time mutual conditioning.

In practice

• Develop more natural conversational avatars.
• Improve real-time human-computer interaction.
• Enhance lip-sync and motion fidelity in virtual agents.

Topics

• Conversational AI
• Full-Duplex Systems
• Speech Synthesis
• Facial Animation
• Flow Matching
• Real-time Generation

Best for: Research Scientist, AI Scientist, NLP Engineer, Computer Vision Engineer

Related on AIssential

• ReFree: Towards Realistic Co-Speech Video Generation via Reward-Free RL and Multilevel Speech Guidance — ReFree-S2V balances precise lip-sync and expressive motion in co-speech video generation using multi-level speech guidance and reward-free RL.
• Temporally-Aligned Evaluation for Audio-Driven Talking Head Generation — Evaluating dynamic generative models requires sequence alignment, not just frame-wise comparison, to handle natural temporal variations.
• TrioPose: Native Triple-Stream Diffusion Transformers for Pose-Guided Text-to-Image Generation — TrioPose natively integrates pose as an independent modality in Diffusion Transformers, enhancing multi-person image generation fidelity.
• IP-Adapter Is All You Need: Towards Fine-Tuning-Free Diffusion-Based Talking Face Generation — Fine-tuning-free talking face generation is achieved by utilizing pretrained Stable Diffusion and IP-Adapter with specialized components.
• LPM 1.0: Video-based Character Performance Model — LPM 1.0 resolves the "performance trilemma" for conversational AI by integrating data, multimodal conditioning, and streaming for real-time, expressive, and stable character video.
• EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement — EntangleCodec unifies audio understanding and generation via caption-aligned semantic-acoustic tokenization.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.