Evaluating and Preserving Lexical Stress in English-to-Chinese Speech-to-Speech Translation

2026-06-13 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new research investigates the underexplored challenge of cross-lingual lexical stress transfer in English-to-Chinese Speech-to-Speech Translation (S2ST), addressing the lack of automatic evaluation metrics for tonal languages. Researchers constructed a stress-annotated Chinese dataset and developed an XLS-R-based Mandarin stress detector. This detector was integrated with the English EmphAssess system to propose a novel objective metric for cross-lingual stress evaluation. Additionally, the team fine-tuned CosyVoice3 to build a stress-aware S2ST system. Experiments demonstrated that this proposed S2ST architecture significantly outperforms existing systems in stress translation capability, while also maintaining competitive overall translation quality. The new evaluation metric also showed a strong correlation with human subjective judgments.

Key takeaway

For NLP Engineers developing English-to-Chinese Speech-to-Speech Translation systems, you should prioritize integrating explicit lexical stress transfer mechanisms. Your current S2ST models likely underperform in conveying emphasis; consider adopting the proposed stress-aware architecture, potentially based on fine-tuning CosyVoice3. Furthermore, utilize the new objective evaluation metric to accurately assess cross-lingual stress preservation, ensuring your system's output maintains speaker intent and naturalness.

Key insights

A novel S2ST system and evaluation metric significantly improve cross-lingual lexical stress transfer from English to Chinese.

Principles

• Lexical stress transfer is vital for S2ST.
• Objective metrics are crucial for tonal languages.
• Stress-aware S2ST systems outperform others.

Method

Constructed a stress-annotated Chinese dataset, developed an XLS-R-based Mandarin stress detector, integrated with EmphAssess for evaluation, and fine-tuned CosyVoice3 for stress-aware S2ST.

In practice

• Use XLS-R for Mandarin stress detection.
• Integrate EmphAssess for cross-lingual stress evaluation.
• Fine-tune CosyVoice3 for stress-aware S2ST.

Topics

• Speech-to-Speech Translation
• Lexical Stress Transfer
• Cross-lingual Evaluation
• Mandarin Stress Detection
• CosyVoice3 Fine-tuning
• XLS-R

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• A Practical Evaluation Method for Long-Form Simultaneous Speech-to-Speech Translation — A new evaluation method for long-form SimulS2ST uses ASR, forced alignment, and sentence embeddings to measure latency and quality.
• Multilingual Emotional Voice Synthesis for Brazilian Portuguese: A Comparative Analysis of Fine-Tuning Approaches — Simpler fine-tuning methods can achieve high-quality multilingual emotional voice synthesis for Brazilian Portuguese.
• Lexical Tone is Hard to Quantize: Probing Discrete Speech Units in Mandarin and Yor\`ub\'a — Current DSU quantization methods struggle to reliably encode lexical tone, prioritizing phonetic structure instead.
• NaturalFlow: Reducing Disruptive Pauses for Natural Speech Flow in Simultaneous Speech-to-Speech Translation — Simultaneous S2ST can achieve natural speech flow by optimizing for fluency and translation quality concurrently, avoiding disruptive pauses.
• Improving quality and robustness in LLM-based text-to-speech systems — LLM-based TTS quality improves through targeted techniques addressing accent leakage, expressiveness, and reliability.
• An Empirical Study on Learning Latent Representations for Emotional Speech Synthesis — Integrating speaker embedding and prosody bottleneck into FastSpeech 2 enables effective emotional speech synthesis and style transfer.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.