Evaluate Clinical ASR Models Faster with Agent Skills and NVIDIA Nemotron Speech

2026-06-09 · Source: NVIDIA Technical Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Medical Devices & Health Technology · Depth: Intermediate, medium

Summary

NVIDIA introduces a clinical Automatic Speech Recognition (ASR) workflow designed to accelerate the evaluation and improvement of speech AI models for specialized medical terminology. This addresses the inherent difficulty in accurately recognizing drug names like Acetaminophen and Cefazolin, procedure names, and diagnoses, which are often missed by general speech systems. The workflow leverages synthetic data generation (SDG) to create pronunciation-aware audio, bypassing the challenges of collecting HIPAA-protected real clinical audio. Key components include NVIDIA agent skills to guide the process, NVIDIA NeMo Data Designer for expanding seed terms into rich datasets, and NVIDIA Magpie TTS Multilingual for synthesizing audio with precise SSML phoneme tags. The system generates a NeMo-compatible JSONL manifest, enabling entity-level ASR evaluation and guiding targeted model adaptation. An orthopedic practice simulation demonstrated the workflow's ability to identify specific error patterns, such as medication name misrecognitions, to inform subsequent improvement cycles.

Key takeaway

For MLOps Engineers tasked with deploying and maintaining clinical ASR systems, you should integrate NVIDIA's agent skill-guided workflow to establish robust, repeatable evaluation cycles. This approach allows you to quickly generate pronunciation-aware synthetic benchmarks, bypassing HIPAA constraints and slow annotation. By focusing on entity-level metrics and incorporating explicit human review for IPA gaps, you can precisely identify and address model weaknesses, ensuring higher accuracy for critical clinical terminology before production deployment.

Key insights

AI agent-guided synthetic data generation with pronunciation control enables rapid, repeatable clinical ASR evaluation and targeted model improvement.

Principles

• Clinical ASR demands pronunciation-accurate, domain-specific data.
• Synthetic data bypasses real clinical audio collection hurdles.
• Repeatable feedback loops drive continuous ASR quality improvement.

Method

The workflow defines a clinical profile, builds a term-centered benchmark, reviews pronunciations, generates synthetic audio via NeMo Data Designer and Magpie TTS, measures ASR behavior, and iteratively refines the model or data.

In practice

• Configure ASR benchmarks using agent skills and clinical profiles.
• Generate pronunciation-aware synthetic audio with NeMo Data Designer.
• Manually review IPA for new or low-confidence clinical terms.

Topics

• Clinical ASR
• Synthetic Data Generation
• NVIDIA Agent Skills
• NeMo Data Designer
• Magpie TTS Multilingual
• Speech AI Evaluation

Code references

• NVIDIA-NeMo/DataDesigner
• NVIDIA/skills
• NVIDIA-NeMo/NeMo

Best for: Machine Learning Engineer, NLP Engineer, MLOps Engineer

Related on AIssential

• Pushing the Limits of On-Device Streaming ASR: A Compact, High-Accuracy English Model for Low-Latency Inference — On-device ASR can achieve high accuracy and low latency on CPU through systematic benchmarking and quantization.
• Combining Real and Synthetic Speech for ASR Adaptation in Brazilian Portuguese — Domain-guided synthetic speech effectively augments real data for ASR adaptation in specialized, low-resource scenarios.
• Real-time voice agents with Stream Vision Agents and Amazon Nova 2 Sonic — Combine Vision Agents with Amazon Nova 2 Sonic and Bedrock for production-ready, real-time voice agents.
• Angelos Perivolaropoulos of ElevenLabs at RAAIS 2026 — Real-world voice AI demands balancing high speech-to-text accuracy with critical low-latency performance for effective interactive systems.
• Corti's new Symphony for Speech-to-Text model beats OpenAI at medical terminology accuracy, highlighting the value of specialized AI — Specialized AI models significantly outperform generalist ones in highly regulated, domain-specific fields like healthcare.
• From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents — Converting text-based tool-calling benchmarks to audio reveals significant model- and task-dependent performance gaps.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by NVIDIA Technical Blog.