MURMUR: An Efficient Inference System for Long-Form ASR

2026-05-31 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Audio and Speech Processing · Depth: Expert, quick

Summary

Murmur is an efficient inference system designed for long-form Automatic Speech Recognition (ASR) that resolves the traditional trade-off between high accuracy and low latency. Existing chunk-based pipelines offer low latency but lose cross-chunk context, while long-context ASR models provide better accuracy but are significantly slower. Murmur operates at two levels: an inter-chunk level, where chunk size is a tunable hyperparameter for balancing accuracy and latency, and an intra-chunk level, which exploits attention sparsity using a sliding window KV cache eviction policy for both output and speech tokens. On the AMI-IHM benchmark, Murmur achieved single-pass accuracy while reducing latency by 4.2x, with additional gains from token eviction causing less than 1% relative tcpWER degradation.

Key takeaway

For Machine Learning Engineers deploying long-form ASR systems, Murmur presents a validated architecture to achieve high accuracy without compromising on latency. You should investigate its two-level inference approach, particularly the tunable chunk sizes and the sliding window KV cache eviction, to optimize your model deployments. This method can significantly improve real-time performance and reduce operational costs for demanding speech processing applications.

Key insights

Murmur efficiently balances long-form ASR accuracy and latency using a two-level inference system.

Principles

• Intermediate chunk sizes optimize ASR accuracy and latency.
• Attention sparsity can be exploited for inference efficiency.
• Sliding window KV cache eviction improves token processing.

Method

Murmur employs a two-level approach: inter-chunk processing with tunable chunk sizes and intra-chunk attention sparsity via a sliding window KV cache eviction policy for output and speech tokens.

In practice

• Tune ASR chunk sizes for specific performance needs.
• Implement KV cache eviction for long-context models.
• Apply attention sparsity techniques to reduce latency.

Topics

• Long-Form ASR
• Inference Systems
• KV Cache Eviction
• Attention Sparsity
• Speech Processing
• Low Latency

Code references

• uw-syfi/Murmur

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Echo: A Joint-Embedding Predictive Architecture for Speaker Diarization and Speech Recognition in a Shared Latent Space — A single 25M-parameter ViT encoder can jointly perform speaker diarization, speech recognition, and source separation in a shared latent space.
• MOSS-Audio Technical Report — MOSS-Audio unifies audio understanding across speech, sound, and music using novel architectural and data strategies.
• PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects — PolySpeech-100 reveals E2E Speech-LLMs excel on dialects, but open-source models struggle with low-resource languages, and CoT prompting can degrade performance.
• CTC: Many Paths, One Word — CTC calculates word probability by summing all valid input-to-output alignments, handling repetitions and blanks.
• Do You Actually Need to Pay for Transcription Software? — Advanced AI transcription and post-processing can be achieved without expensive subscriptions by leveraging free or existing tools.
• Beyond the Mouth: Upper-Face Affective Cues in Audiovisual Sentence Recognition under Acoustic Uncertainty — Upper-face affective cues enhance audiovisual speech recognition robustness and confidence under acoustic uncertainty, beyond mouth-region information.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.