Contextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wild

2026-04-10 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, long

Summary

Argmax, Inc. and UCLA introduce Contextual Earnings-22, an open dataset designed to standardize benchmarking for contextual speech-to-text (STT) systems. This dataset, built upon Earnings-22, addresses the plateauing accuracy of STT on academic benchmarks by focusing on custom vocabulary recognition, which is critical for real-world usability. Contextual Earnings-22 features 760 context-dense 15-second audio clips from earnings calls, paired with manually reviewed transcripts and realistic custom vocabulary contexts, including person, company, and product names. It supports evaluation in two scenarios: local context (precise, no distractors) and global context (realistic, with distractors). The researchers established six strong baselines using both keyword prompting and keyword boosting methods, demonstrating significant improvements in contextual term recognition, though robustness to distractors remains a key differentiator.

Key takeaway

For AI Engineers and Research Scientists developing or deploying speech-to-text systems, Contextual Earnings-22 provides a crucial public benchmark to assess real-world performance. You should utilize this dataset to rigorously evaluate your models' ability to handle custom vocabularies and measure their robustness against distractors, which is often overlooked by traditional WER metrics. This will help you identify systems that truly excel in high-stakes, context-dependent applications like earnings call transcription.

Key insights

Contextual Earnings-22 offers a standardized benchmark for evaluating speech-to-text systems' custom vocabulary recognition in realistic settings.

Principles

• Custom vocabulary accuracy is critical for real-world STT utility.
• Contextual conditioning significantly improves custom term recognition.
• Robustness to distractors differentiates STT system performance.

Method

The Contextual Earnings-22 pipeline extracts contextual keywords using GPT-5, segments transcripts, performs forced alignment with wav2vec, and manually reviews/corrects data to create 15-second audio clips with local and global context lists.

In practice

• Use keyword-centric metrics alongside WER for STT evaluation.
• Test STT systems with both precise and noisy context lists.
• Consider distractor robustness when selecting STT solutions.

Topics

• Speech Recognition Benchmarks
• Contextual Speech-to-Text
• Custom Vocabulary
• Earnings-22 Dataset
• Keyword Prompting

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India — Real-world ASR performance in India requires unscripted data, orthographically-informed metrics, and regional analysis to overcome current benchmark limitations.
• The Acoustic Camouflage Phenomenon: Re-evaluating Speech Features for Financial Risk Prediction — Media-trained vocal regulation in corporate calls degrades acoustic feature utility for financial risk prediction.
• Perceptual compensation for tonal context in self-supervised speech models — Self-supervised speech models like wav2vec2.0 may require supervised fine-tuning to achieve perceptual compensation for complex phonological features like Mandarin tones.
• Scaling Human and G2P Supervision for Robust Phonetic Transcription — G2P supervision for phonetic transcription has diminishing returns beyond 20-30 hours of human annotation.
• Adding Benchmaxxer Repellant to the Open ASR Leaderboard — Private datasets enhance ASR leaderboard trustworthiness by preventing benchmaxxing and revealing nuanced model performance.
• Scale AI launches Voice Showdown, the first real-world benchmark for voice AI — and the results are humbling for some top models — Scale AI has launched Voice Showdown, a new global, human-preference-based arena designed to benchmark voice AI models using real human interaction across over 60 languages, addressing the shortcoming…

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.