Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing

2026-07-21 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Modern Automatic Speech Recognition (ASR) models, trained on heterogeneously annotated data, treat transcription style (verbatim vs. intended) as an uncontrolled latent variable. This leads to measurable decoding instability, evaluation confounding, with up to 60% of reported Word Error Rate (WER) attributable to style mismatch, and unreliable word-level timing. Researchers demonstrate that ASR models already encode both styles, and the primary challenge is controlled activation. By employing coverage-aware decoder task tokens, trained on parallel verbatim/intended transcript pairs, they achieved a zero-shot German disfluency F1 increase from 10% to 79%, despite English-only training. Full English-only fine-tuning further surpassed all baselines in verbatim accuracy, disfluency detection, and intended-mode quality across both languages. Additionally, supervised cross-attention fine-tuning improved word-level timestamps on disfluent speech beyond forced-alignment baselines. A new task, "verbatimize", is also proposed for scalable creation and enrichment of high-quality canonical verbatim speech corpora.

Key takeaway

For NLP engineers developing or evaluating ASR systems, this research offers a critical method to address transcription style inconsistencies. Your systems can achieve significantly higher verbatim accuracy and disfluency detection by integrating coverage-aware decoder task tokens. Consider adopting the "verbatimize" task to build higher-quality, style-controlled speech corpora, thereby reducing evaluation confounding and improving word-level timing precision in your ASR deployments.

Key insights

ASR models encode both verbatim and intended transcription styles; controlled activation is the key challenge.

Principles

• ASR style mismatch confounds evaluation, impacting up to 60% of WER.
• Models inherently encode multiple transcription styles, requiring activation control.

Method

Utilize coverage-aware decoder task tokens with parallel verbatim/intended pairs, and supervised cross-attention fine-tuning to improve word-level timing on disfluent speech.

In practice

• Implement coverage-aware task tokens for ASR style control.
• Use the "verbatimize" task to create verbatim speech corpora.

Topics

• Automatic Speech Recognition
• Transcription Style
• Disfluency Detection
• Word-Level Timing
• Decoder Task Tokens
• Speech Corpora
• Verbatim Transcriptions

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Customizing ASR for Language Documentation and Resource Prioritization — Prioritize transcription correction over manual alignment for ASR fine-tuning in language documentation.
• Suppressing Unnecessary Clarification Requests for Unknown Word Acquisition in Spoken Dialogue Using Syllable-Based ASR Confidence — Integrating ASR and segmentation confidence into dialogue policy effectively reduces unnecessary clarification requests for unknown words.
• On the Role of Conversational Timing in Synthetic Training Data for ASR — Conversational timing, specifically overlap and gap characteristics, significantly impacts ASR performance in synthetic training data.
• Analyzing Error Propagation in Korean Spoken QA with ASR-LLM Cascades — ASR-stage information loss, especially single-character errors in Korean, is the primary bottleneck for ASR-LLM cascade performance.
• Clicky Meets Universal-3.5: Voice-Navigate Your Code — Dynamic key term updating from screen content significantly enhances speech-to-text accuracy for domain-specific vocabulary, enabling precise voice navigation.
• Scaling Human and G2P Supervision for Robust Phonetic Transcription — G2P supervision for phonetic transcription has diminishing returns beyond 20-30 hours of human annotation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.