How I Built a Custom Pronunciation Coach

2026-07-18 · Source: NLP on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, short

Summary

A software engineer developed Lingue, an open-source, L1-aware pronunciation coach powered by "viennounce-core", to overcome limitations of commercial apps like Duolingo and ELSA Speak. This custom tool provides private, customizable phonetic feedback, specifically targeting L1 transfer errors common among Vietnamese English learners, such as dropping final consonants. The project achieved near-production-grade ASR capabilities without collecting new data, utilizing a 2-hour T4 / 1-hour A10G serverless GPU session on Modal's free tier for training. Initial "Wav2Vec2 Forced Aligner" models scored native speech at 84.3% due to phonetic overlaps. Through dynamic calibration with the L2-ARCTIC Vietnamese speaker dataset, native speaker scores improved to 95.7%. A critical enhancement involved expanding the "PhoneticASRModel"'s classification head from 60 to 392 "espeak" tokens, resolving "modulo-60 collisions" and boosting Recall (True Error Detection) from 28.9% to 40.3% on the L2-ARCTIC validation suite.

Key takeaway

For Machine Learning Engineers developing custom language learning applications, you can build highly accurate, L1-aware pronunciation coaches without extensive proprietary datasets or large budgets. Consider utilizing open-source models like Wav2Vec2 and serverless GPU platforms such as Modal to achieve production-grade results. This approach allows for deep customization and privacy, enabling you to address specific phonetic transfer errors and provide precise, actionable feedback beyond generic commercial offerings.

Key insights

A custom, L1-aware pronunciation coach can be built affordably using open-source models and serverless GPUs, achieving high accuracy.

Principles

• Open-source models enable advanced tools without large budgets.
• L1-aware phonetic analysis improves error detection for specific learners.
• Serverless GPU platforms offer cost-effective model training.

Method

The method involved using "sea-g2p" for text-to-IPA, "Wav2Vec2 Forced Aligner" for audio-phoneme alignment and GOP scores, dynamic calibration with L2-ARCTIC data, and expanding the ASR model's classification head to 392 "espeak" tokens.

In practice

• Use "viennounce-core" for custom pronunciation feedback.
• Utilize Modal for cost-effective GPU training.
• Expand ASR classification heads to resolve token collisions.

Topics

• Pronunciation Coaching
• L1 Transfer Errors
• Wav2Vec2
• Serverless GPU
• Phonetic ASR
• Open-Source AI
• Modal

Code references

• phuocnguyen90/vienounce-core

Best for: Software Engineer, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Pretrained self-supervised speech models can recognize unseen consonants — Pretrained self-supervised speech models effectively generalize to and accurately recognize typologically rare phonemes, such as click consonants.
• FlowEdit: Associative Memory for Lifelong Pronunciation Adaptation in Flow-Matching TTS — FlowEdit enables lifelong TTS pronunciation adaptation via latent embedding edits stored in a Hopfield Network, reducing errors by 92.7%.
• A New Chapter for Realtime AI: Reasoning, Translation, and Real-Time Transcription — New OpenAI models enhance real-time AI voice applications with advanced translation, transcription, and reasoning capabilities.
• ASTRA: A Scalable Next-Generation ATCO Training Simulator with Autonomous Simpilots — Localized speech models and hybrid AI are critical for scalable, objective ATCO training simulation.
• Phonetic Error Analysis of Raw Waveform Acoustic Models — Raw waveform acoustic models achieve state-of-the-art TIMIT PER, with error patterns revealing specific phonetic class improvements from BLSTMs and transfer learning.
• From A to B to A: Palindromic Zero-Shot Voice Conversion with Non-Parallel Data — A voice conversion framework uses KNN over WavLM to create synthetic training pairs from non-parallel data for zero-shot, multilingual VC.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by NLP on Medium.