Unified Audio Intelligence Without Regressing on Text Intelligence

2026-07-06 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Nemotron-Labs-Audex-30B-A3B (Audex) is a new unified audio-text large language model, developed from the Nemotron-Cascade-2-30B-A3B text-only MoE LLM. Audex employs a simple architecture featuring a single Transformer decoder, where audio inputs are encoded and projected into the text embedding space, and both text and quantized audio output tokens are processed uniformly during generation. This design facilitates robust audio-text fusion, seamless multimodal generation, and integration with existing LLM infrastructure. Training involved 157.4 billion audio tokens and 320.5 billion text tokens, utilizing multi-stage supervised training, text-only Cascade RL, and multi-domain on-policy distillation. Audex achieves high performance across audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech generation, while maintaining the strong reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only foundation with minimal regression. The model checkpoints are released for open research.

Key takeaway

For AI scientists and ML engineers developing multimodal LLMs, Audex demonstrates a viable path to unified audio-text intelligence. You can achieve strong audio understanding and generation capabilities, including speech-to-speech, without sacrificing the advanced reasoning and long-context abilities of text-only models. Consider exploring the released model checkpoints to integrate robust audio processing directly into your existing LLM infrastructure.

Key insights

Audex unifies audio and text intelligence in a single LLM without compromising text capabilities.

Principles

• Unified decoder simplifies multimodal LLMs.
• Projecting audio to text embedding space works.
• Multi-stage training enhances multimodal performance.

Method

Audex uses a single Transformer decoder, encoding audio inputs into the text embedding space. It treats text and quantized audio output tokens uniformly during generation, trained with multi-stage supervised learning, Cascade RL, and on-policy distillation.

In practice

• Generate speech from text directly.
• Perform speech-to-speech translation.
• Integrate audio processing into LLM workflows.

Topics

• Multimodal LLMs
• Audio Intelligence
• Speech Recognition
• Text-to-Speech
• Transformer Decoders
• Nemotron-Labs-Audex

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Introducing SAM Audio: The First Unified Multimodal Model for Audio Separation | AI at Meta — SAM Audio unifies multimodal audio separation using text or visual prompts for diverse applications.
• Improving quality and robustness in LLM-based text-to-speech systems — LLM-based TTS quality improves through targeted techniques addressing accent leakage, expressiveness, and reliability.
• HybridCodec: Fast Dual-Stream, Semantically Enhanced Neural Audio Codec — HybridCodec unifies dual-stream and SSL distillation for efficient, semantically enhanced neural audio encoding.
• Code Switching in Real-Time | Universal-Streaming Speech-to-Text — Assembly AI's universal model enables real-time, zero-latency code-switched speech-to-text across six languages.
• Deepgram speech-to-text and voice models now available natively on Together AI — Real-time voice agents require integrated STT, LLM, and TTS on a single platform to minimize latency and operational complexity.
• Acoustic Prompting via Stage-wise Modulation for Few-Shot Learning in Audio Language Models — Integrating trainable audio-side prompts with text-side prompt tuning significantly enhances few-shot adaptation in Audio-Language Models.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.