Using embeddings to predict spoken word duration and pitch in Mandarin monosyllabic words

2026-07-02 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

A recent study demonstrates that contextualized embeddings (CEs) can predict spoken word duration for Mandarin monosyllabic CV words, building on previous findings that CEs predict time-normalized f0 contours. Analyzing 7470 tokens extracted from a Mandarin spontaneous speech corpus, the research shows CEs are predictive for duration above chance, both at the type and individual token levels, as confirmed by type-wise and token-wise permutation baselines. Furthermore, the predicted durations are precise enough to accurately back-transform predicted f0 contours from a [0,1] normalized time scale to a millisecond time scale. These resulting predicted contours closely approximate empirical contours and also surpass a permutation baseline.

Key takeaway

For NLP Engineers developing Mandarin speech synthesis or prosody modeling tools, this research indicates that integrating contextualized embeddings can significantly improve prediction accuracy for both spoken word duration and pitch. You should consider leveraging these embeddings to generate more natural-sounding synthetic speech and enhance the precision of prosodic feature analysis. This approach offers a robust method for transforming normalized f0 contours to real-time scales.

Key insights

Contextualized embeddings effectively predict both duration and pitch contours for Mandarin monosyllabic words.

Principles

• Contextualized embeddings predict speech prosody.
• Prediction works at both type and token levels.
• Predicted durations enable accurate f0 contour back-transformation.

Method

The study extracted 7470 Mandarin monosyllabic CV word tokens from spontaneous speech, then used contextualized embeddings to predict duration and back-transform f0 contours.

In practice

• Improve synthetic speech naturalness.
• Enhance speech recognition models.
• Analyze prosodic features automatically.

Topics

• Contextualized Embeddings
• Mandarin Speech
• Speech Prosody
• Word Duration Prediction
• Pitch Prediction
• Computational Linguistics

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• Lexical Tone is Hard to Quantize: Probing Discrete Speech Units in Mandarin and Yor\`ub\'a — Current DSU quantization methods struggle to reliably encode lexical tone, prioritizing phonetic structure instead.
• An Empirical Study on Learning Latent Representations for Emotional Speech Synthesis — Integrating speaker embedding and prosody bottleneck into FastSpeech 2 enables effective emotional speech synthesis and style transfer.
• Live demo of AssemblyAI's Universal-3.5 Pro Realtime Speech-to-Text model — Contextual prompting and noise suppression significantly enhance real-time speech-to-text accuracy across diverse languages and challenging audio environments.
• PASQA: Pitch-Accent-Focused Speech Quality Assessment Model Trained on Synthetic Speech with Accent Errors — PASQA accurately assesses pitch-accent correctness in synthetic speech by leveraging a specialized dataset and architectural enhancements.
• A Theoretical Framework for Acoustic Neighbor Embeddings — A theoretical framework provides a probabilistic interpretation of acoustic neighbor embedding distances, simplifying phonetic similarity measurement.
• From Tokens to Faces: Investigating Discrete Speech Representations for 3D Facial Animation — Encoding phonetic classes in discrete speech representations improves 3D facial animation accuracy.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.