RedVox: Safety and Fairness Gaps in Speech Models Across Languages

2026-06-25 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

The RedVox benchmark addresses significant safety and fairness gaps in speech-capable models, particularly beyond English and under naturalistic conditions. A survey of leading speech model releases revealed that only 8% document any multilingual safety analysis. RedVox, a new multilingual safety and fairness benchmark, is built on real voices and covers unsafe and unfair stereotypical requests across five languages: English, French, Italian, Spanish, and German. Evaluation of eight prominent models using RedVox found that vulnerabilities persist even in non-adversarial settings, are exacerbated in non-English languages, and are amplified when requests originate from spoken input. The research also highlights unique personal and privacy challenges associated with collecting naturalistic speech data from human participants.

Key takeaway

For NLP Engineers deploying speech-capable models globally, you must prioritize multilingual safety and fairness evaluations. Your current models likely have significant vulnerabilities in non-English languages, especially when processing spoken input, as only 8% of leading releases document such analysis. You should integrate benchmarks like RedVox into your testing pipeline to identify and mitigate these risks, ensuring equitable and safe model performance across diverse linguistic contexts.

Key insights

Speech models exhibit significant safety and fairness vulnerabilities, especially in non-English languages and with spoken input.

Principles

• Multilingual safety analysis is critically under-documented.
• Spoken input amplifies model vulnerabilities.
• Naturalistic speech data collection poses privacy challenges.

Method

RedVox evaluates speech models by using real voices to test unsafe/unfair stereotypical requests across five languages.

In practice

• Test speech models with spoken inputs in multiple languages.
• Prioritize privacy in naturalistic speech data collection.

Topics

• Speech Models
• Multilingual AI
• AI Safety
• AI Fairness
• RedVox Benchmark
• Speech Data Privacy

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, NLP Engineer, AI Ethicist

Related on AIssential

• Culturally-Adapted Red-Teaming Across East and Southeast Asian Contexts: A Methodological and Comparative Analysis — Direct translation of safety benchmarks fails to capture cultural context, underestimating LLM risks in multilingual settings.
• Transcribing speech is never neutral. It shapes power and bias — Language technologies trained on prestige dialects often misrepresent non-standard and Indigenous communication, leading to serious consequences.
• The Inattentional Gap: Task-Conditioned Language and Vision Models Omit the Safety-Critical Signals They Can Otherwise Report — Task-conditioned AI models exhibit an "Inattentional Gap," failing to report safety-critical signals they can otherwise detect when unconstrained.
• The Multilingual AI Gap Is Not Closing. It Is Being Rebranded. — Linguistic inclusion in AI requires community control and governance, not just more data or better benchmark scores.
• Low-Resource Safety Failures Are Action Failures, Not Representation Failures — Low-resource safety failures stem from decision calibration, not representation transfer, offering a recalibration solution.
• VoxSafeBench: Not Just What Is Said, but Who, How, and Where — SLMs struggle with social alignment when critical safety, fairness, or privacy cues are conveyed acoustically.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.