Light-weight Pronunciation Assessment via Discrete Speech Token Surprisal

2026-06-19 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Speech Processing · Depth: Expert, long

Summary

A new lightweight pronunciation assessment framework is introduced, designed to operate with minimal or no labeled learner data by training exclusively on native speech resources. The system discretizes learner speech using a self-supervised learning (SSL) encoder and a K-means codebook, then employs a token language model (TLM) to calculate surprisal, indicating phonotactic deviations. An optional transcript-guided Text2DUnit–DTW module predicts canonical native token sequences from reference text and aligns them with acoustic tokens to generate error-sensitive features. These surprisal and alignment features are combined via simple regression. On the SpeechOcean762 dataset, the framework achieved a Pearson Correlation Coefficient (PCC) improvement from 0.60 to 0.66 with transcript guidance, approaching supervised baseline performance. Cross-dataset evaluation on L2-ARCTIC also demonstrated consistent gains, and the system proved robust even with an order-of-magnitude reduction in native training data.

Key takeaway

For Machine Learning Engineers or NLP Engineers building pronunciation assessment systems, especially for low-resource languages or specialized speaking styles, this framework offers a compelling solution. You can achieve competitive performance by training solely on native speech, significantly reducing the need for costly labeled learner data, phoneme inventories, or forced alignment. Implement this approach to develop robust computer-assisted language learning tools more efficiently, leveraging transcript guidance for enhanced accuracy and generalization across diverse datasets.

Key insights

Lightweight pronunciation assessment uses native speech token surprisal and text-guided alignment, avoiding costly labeled learner data.

Principles

• Native speech defines phonotactic norms.
• Discrete tokens abstract acoustic space.
• Surprisal measures phonotactic deviation.

Method

Discretize speech via SSL encoder and K-means. Train n-gram TLM on native tokens. Predict native token sequences from text using Text2DUnit. Align learner acoustic tokens with text-derived tokens via DTW. Fuse surprisal and alignment features with regression.

In practice

• Use HuBERT base Layer 9 for SSL features.
• Fit K-means codebook with K=512.
• Train a 3-gram token language model.

Topics

• Pronunciation Assessment
• Discrete Speech Tokens
• Self-supervised Learning
• Token Surprisal
• Dynamic Time Warping
• Computer-Assisted Language Learning

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Light-weight Pronunciation Assessment via Discrete Speech Token Surprisal — Lightweight pronunciation assessment can be achieved by analyzing discrete speech token surprisal from native speech models, reducing reliance on costly labeled non-native data.
• L-Proto: Language-Aware Episodic Prototypical Training for Multilingual Speaker Verification — L-Proto improves multilingual speaker verification by using language-consistent episodic training to disentangle speaker identity from linguistic characteristics.
• Phonikud: Overcoming Phonetic Underspecification for Hebrew Text-To-Speech — Fully specifying phonetic features like stress significantly improves Hebrew TTS accuracy and enables smaller models.
• Phonological Subspace Collapse Is Aetiology-Specific and Cross-Lingually Stable: Evidence from 3,374 Speakers — Phonological subspace analysis offers a robust, training-free, and cross-lingually stable method for dysarthria characterization.
• Lexical Tone is Hard to Quantize: Probing Discrete Speech Units in Mandarin and Yor\`ub\'a — Current DSU quantization methods struggle to reliably encode lexical tone, prioritizing phonetic structure instead.
• FlowEdit: Associative Memory for Lifelong Pronunciation Adaptation in Flow-Matching TTS — FlowEdit enables lifelong TTS pronunciation adaptation via latent embedding edits stored in a Hopfield Network, reducing errors by 92.7%.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.