Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs

2026-07-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, extended

Summary

Lychee-FD, a novel end-to-end full-duplex framework, addresses critical modality interference in Spoken Language Models (SLMs) that causes knowledge degradation and unnatural interactions. An in-depth analysis revealed that this interference stems from inherent gradient conflicts between acoustic and semantic modeling in shared deep parameter spaces, manifesting as optimization divergence and semantic dilution. Lychee-FD mitigates these issues through a hierarchical parameter separation strategy for deep layers and a semantic alignment channel to maintain cross-modality coherence. Extensive experiments show Lychee-FD significantly advances the state of the art, achieving a 7.4% improvement on Spoken QA, a 28.5% gain on FullDuplexBench 1.5, and the highest UTMOS score of 4.50, all while maintaining ultra-low latency.

Key takeaway

For Machine Learning Engineers developing full-duplex Spoken Language Models, Lychee-FD's approach offers a robust solution to modality interference. You should consider implementing hierarchical parameter separation for deep layers and a semantic alignment channel to resolve gradient conflicts and semantic dilution. This strategy enables superior speech intelligence and interaction fluidity, as demonstrated by Lychee-FD's 7.4% Spoken QA improvement and 28.5% FullDuplexBench 1.5 gain, without compromising inference efficiency.

Key insights

Modality interference in full-duplex SLMs arises from deep-layer gradient conflicts and semantic dilution.

Principles

• Deep layers in SLMs exhibit conflicting acoustic and semantic gradients.
• Sparse text alignment dilutes semantic supervision, degrading knowledge retention.
• Decoupling modalities can enhance both SLM intelligence and interaction fluidity.

Method

Lychee-FD employs hierarchical parameter separation for deep layers and a semantic alignment channel with continuous textual supervision to resolve gradient conflicts and semantic dilution.

In practice

• Separate deep layers for acoustic and semantic processing in multi-modal models.
• Utilize continuous textual supervision to anchor semantic robustness.
• Implement Directed Acyclic Graph Pipeline Parallelism (DAG-PP) for efficient multi-head inference.

Topics

• Full-Duplex SLMs
• Modality Interference
• Hierarchical Parameter Separation
• Semantic Alignment Channel
• Spoken Question Answering
• Real-Time Inference
• Gradient Conflicts

Code references

• HITsz-TMG/Lychee-FD

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition — Partial Information Decomposition (PID) quantifies unique, redundant, and synergistic modality contributions in MLLMs to understand interaction.
• VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents — VideoFDB is the first benchmark to evaluate full-duplex audio-visual conversational agents, revealing current systems' limitations in nonverbal grounding.
• LoMo: Local Modality Substitution for Deeper Vision-Language Fusion — LoMo addresses VLM "carrier sensitivity" by creating semantically equivalent text and image representations through local modality substitution.
• Bridging Modal Isolation in Interleaved Thinking: Supervising Modality Transitions via Stepwise Reinforcement — Effective interleaved multimodal reasoning requires explicit structural supervision at modality boundaries, not just end-task optimization.
• MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models — MTR-DuplexBench offers a comprehensive, multi-round evaluation for Full-Duplex Speech Language Models.
• Multi-Granularity Reasoning for Natural Language Inference — MGRN enhances NLI by explicitly modeling hierarchical semantic interactions across transformer layers.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.