Adding Benchmaxxer Repellant to the Open ASR Leaderboard

2026-05-04 · Source: Hugging Face - Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, medium

Summary

The Hugging Face Open ASR Leaderboard, launched in September 2023, has introduced new private datasets from Appen Inc. and DataoceanAI to combat "benchmaxxing" and improve the trustworthiness of its evaluations. These high-quality English ASR datasets cover both scripted and conversational speech across multiple accents, including Australian, Canadian, Indian, American, and British English. While the datasets are private to prevent test-set contamination, the leaderboard allows users to optionally include them in the average Word Error Rate (WER) calculation via a toggle. The default average WER continues to be computed only on public datasets. This update aims to provide a more holistic view of ASR performance by highlighting model strengths and weaknesses across diverse conditions, such as scripted vs. conversational styles and American vs. non-American accents.

Key takeaway

For AI Engineers and Research Scientists evaluating ASR models, leverage the Open ASR Leaderboard's new private datasets to gain a more robust understanding of model performance beyond public benchmarks. Actively use the "Private data" toggle to assess how models perform on diverse accents and conversational speech, which can reveal critical gaps and biases not apparent in standard evaluations. This approach helps identify models truly optimized for real-world, varied audio conditions.

Key insights

Private datasets enhance ASR leaderboard trustworthiness by preventing benchmaxxing and revealing nuanced model performance.

Principles

• Standardization is crucial for meaningful benchmarking.
• Openness fosters community improvement and contributions.
• Private datasets increase evaluation trustworthiness.

Method

New private datasets are integrated into the Open ASR Leaderboard, with an optional toggle to include them in WER calculations. Evaluation scripts and UI code remain open-sourced, and a normalizer standardizes model outputs.

In practice

• Use the leaderboard's toggle to include private datasets.
• Submit models via GitHub PR for private evaluation.
• Self-report public metrics in model cards for unverified listing.

Topics

• Open ASR Leaderboard
• Automatic Speech Recognition
• Benchmarking
• Dataset Contamination
• Appen Inc.

Code references

• huggingface/blog
• huggingface/open_asr_leaderboard
• openai/whisper

Best for: AI Engineer, Research Scientist, Machine Learning Engineer, AI Scientist, NLP Engineer

Related on AIssential

• Introducing the FFASR Leaderboard: Benchmarking ASR in the Real World — Far-field ASR performance significantly degrades in realistic acoustic conditions, a gap the FFASR Leaderboard quantifies.
• Contextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wild — Contextual Earnings-22 offers a standardized benchmark for evaluating speech-to-text systems' custom vocabulary recognition in realistic settings.
• Subtitle-Aligned Fine-Tuning of Whisper for Swiss German ASR: Benchmark Contamination, Convention Mismatch, and an Honest Baseline at 25.6% WER (13.8% cWER) — Benchmark contamination significantly inflates reported ASR performance, especially for dialectal variations.
• Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India — Real-world ASR performance in India requires unscripted data, orthographically-informed metrics, and regional analysis to overcome current benchmark limitations.
• Evaluation of Automatic Speech Recognition Using Generative Large Language Models — Generative LLMs significantly improve ASR evaluation by aligning better with human perception than traditional WER.
• CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks — CoEval enables trustworthy, task-specific language model ranking through ensemble self-evaluation, eliminating external labels or benchmarks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Hugging Face - Blog.