Gradient-Based Speech-to-Text Alignment for Any ASR Model: From CTC to Speech LLMs

2026-07-09 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Speech Recognition · Depth: Expert, extended

Summary

Researchers propose a generic gradient-based alignment method for speech-to-text alignment, applicable to any differentiable Automatic Speech Recognition (ASR) model, including Connectionist Temporal Classification (CTC), transducer, attention-based encoder-decoders (AED), and speech Large Language Models (LLMs). This technique computes the gradient of each teacher-forced token's log probability relative to the input, converting it into a per-frame saliency matrix, which is then decoded into word boundaries via dynamic programming. The method requires no training or model modifications and aligns on the high-resolution input grid. Evaluated across sixteen models from four families on TIMIT and Buckeye datasets, it consistently yields usable alignments. While generally competitive with native aligners, it outperforms them for streaming models and Canary-Qwen, and even surpasses Whisper-large-v3's cross-attention at optimal encoder depths. Its primary drawback is the computational cost of one backward pass per token.

Key takeaway

For Machine Learning Engineers or AI Scientists requiring precise speech-to-text alignment, especially with models lacking native aligners or in streaming contexts, you should consider gradient-based alignment. This method provides a universal, training-free approach that often surpasses weak native aligners and offers input-grid resolution. Be aware of the increased computational cost due to the per-token backward pass, which makes it less practical for high-throughput scenarios.

Key insights

A gradient-based method provides universal, training-free speech-to-text alignment across all differentiable ASR models, often outperforming weak native aligners.

Principles

• Gradient saliency can reveal precise temporal alignment.
• Input-grid alignment offers higher resolution than encoder-grid.
• Character targets improve gradient alignment for autoregressive models.

Method

Calculate gradient of teacher-forced token log probability w.r.t. input, reduce to per-frame saliency, then decode into word boundaries using dynamic programming. This needs no training or model modification.

In practice

• Apply gradient-based alignment to ASR models lacking native aligners.
• Use character targets for autoregressive models for better accuracy.
• Experiment with intermediate encoder layers for optimal gradient signal.

Topics

• Speech-to-Text Alignment
• Gradient-Based Methods
• ASR Models
• Speech LLMs
• Dynamic Programming
• Saliency Maps

Best for: AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Gradient-Based Speech-to-Text Alignment for Any ASR Model: From CTC to Speech LLMs — A universal gradient-based method aligns speech-to-text for any ASR model, including LLMs, with high precision and no retraining.
• Phonetic forced alignment for low-resource language varieties: Model training and evaluation on Chengdu Mandarin — A bootstrapping pipeline enables accurate phonetic forced alignment for low-resource language varieties using pseudo-labeling and fine-tuning.
• AST: Adaptive, Seamless, and Training-Free Precise Speech Editing — AST enables training-free, precise speech editing by manipulating latent spaces of pre-trained autoregressive TTS models.
• An Omnilingual-ASR-Based Speech-LLM System for the 2nd MLC-SLM Challenge — A cascaded speech-LLM system for omnilingual ASR leverages embedding-based clustering and LoRA adaptation for robust performance.
• NAVER LABS Europe Submission to the Instruction-following 2026 Short Track — The system combines an improved speech projector and synthetic domain-specific data for superior instruction-following speech processing.
• ART: A Faster Way to Train Speech Recognition Models — ART accelerates RNN-T training by combining CTC for baseline alignment with residual learning for refinement.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.