The case for stakeholder-driven AI auditing in automatic speech recognition

2026-03-16 · Source: Nature Machine Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, quick

Summary

A correspondence published in Nature Machine Intelligence on March 16, 2026, argues for stakeholder-driven AI auditing, particularly in automatic speech recognition (ASR) systems. The authors highlight a growing need to assess AI system performance to prevent potential harms as these systems become infrastructural. While AI audits typically compare a system's output to a "ground truth," the article challenges the notion of a single, definitive ground truth, especially when AI interprets complex human realities. For ASR systems, where ground truth is a text transcription of speech, variations like including or excluding stutters and filler words (e.g., "um," "uh") can both be considered valid transcriptions, complicating the auditing process. This complexity necessitates a re-evaluation of how ground truth is defined and applied in AI auditing.

Key takeaway

For AI scientists and NLP engineers developing or auditing ASR systems, recognize that the "ground truth" for speech transcription is not singular. Your auditing methodologies should incorporate stakeholder perspectives to define appropriate nominal values, accounting for linguistic nuances like filler words or stutters. This approach ensures audits reflect real-world usage and prevent misinterpretations of system performance, leading to more robust and equitable AI deployments.

Key insights

Defining "ground truth" in AI auditing is complex, especially for systems interpreting nuanced human data like speech.

Principles

• AI audits compare nominal vs. actual system values.
• Single ground truth is challenged for complex AI.
• Multiple valid transcriptions exist for speech.

In practice

• Consider diverse transcription norms for ASR.
• Involve stakeholders in defining ASR ground truth.

Topics

• AI Auditing
• Automatic Speech Recognition
• Ground Truth
• Stakeholder-Driven AI
• Generative AI

Best for: AI Scientist, Research Scientist, NLP Engineer, AI Researcher, AI Ethicist, Policy Maker

Related on AIssential

• Ground truth is a process, not a dataset — Ground truth for complex AI evaluation must be an evolving process, not a static dataset.
• Transcribing speech is never neutral. It shapes power and bias — Language technologies trained on prestige dialects often misrepresent non-standard and Indigenous communication, leading to serious consequences.
• AI Doesn’t Need to Be Right. It Only Needs to Sound Procedural — AI's procedural language masks uncertainty, enabling policy formation without visible accountability beyond mere hallucination.
• Adaptive auditing of AI systems with anytime-valid guarantees — Adaptive AI auditing can achieve statistical rigor using dueling hypotheses and anytime-valid e-processes, even with small, flexible datasets.
• AI in audit: The gap between knowing and doing — Audit firms must prioritize AI governance and cultivate auditor skepticism over mere deployment to realize true value.
• A Method From 1979 Beat the Algorithm I Was Most Proud Of. I Left the Proof in My Own Paper. — Simpler models and honest evaluation often outperform complex AI systems, especially for token-based data.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Nature Machine Intelligence.