CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding

2026-06-03 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

CleanCodec is a novel denoising audio codec designed for efficient and robust speech tokenization, addressing the challenge of balancing reconstruction quality with token efficiency in neural audio codecs. It reframes audio tokenization as a selective information bottleneck, learning to encode only perceptually important features while discarding imperceptible background noise and recording artifacts. Operating at just 12.5 tokens per second, CleanCodec achieves state-of-the-art tokenization efficiency, significantly improving speaker similarity and speech intelligibility compared to existing codecs. Furthermore, evaluations show improved performance and up to 17x faster inference in downstream text-to-speech and voice conversion tasks, demonstrating substantial efficiency gains for speech processing pipelines.

Key takeaway

For machine learning engineers optimizing speech processing pipelines, CleanCodec offers a compelling solution to enhance both efficiency and quality. You should consider integrating CleanCodec to achieve state-of-the-art tokenization efficiency at 12.5 tokens per second, improving speaker similarity and speech intelligibility. This can lead to up to 17x faster inference and better performance in your text-to-speech and voice conversion applications.

Key insights

CleanCodec reframes audio tokenization as a selective information bottleneck, encoding only perceptually important speech features.

Principles

• Encoding perceptually important features improves token efficiency and reconstruction quality.
• Discarding imperceptible information enhances linguistic and acoustic meaning.

Method

CleanCodec operates as a denoising audio codec, learning to selectively encode perceptually important features and discard noise and artifacts.

In practice

• Apply CleanCodec in text-to-speech systems for improved performance and faster inference.
• Integrate CleanCodec into voice conversion pipelines to enhance quality and speed.

Topics

• Audio Codecs
• Speech Tokenization
• CleanCodec
• Denoising
• Text-to-Speech
• Voice Conversion
• Perceptual Encoding

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Text-Utilization for Encoder-dominated Speech Recognition Models — Efficient text-only data integration improves encoder-dominated speech recognition, favoring simpler models and larger encoders.
• Introducing Post-Stream Refinement: Higher-Accuracy Real-Time Transcription — Post-Stream Refinement enhances real-time speech recognition by combining instant partial results with a higher-accuracy, context-aware second pass.
• Lexical Tone is Hard to Quantize: Probing Discrete Speech Units in Mandarin and Yor\`ub\'a — Current DSU quantization methods struggle to reliably encode lexical tone, prioritizing phonetic structure instead.
• CTC: Many Paths, One Word — CTC calculates word probability by summing all valid input-to-output alignments, handling repetitions and blanks.
• LoSATok: Low-dimensional Semantic-Acoustic Tokenizer for Cross-Domain Audio Understanding and Generation — LoSATok unifies audio understanding and generation via low-dimensional semantic-acoustic tokens, reducing Diffusion Transformer modeling complexity.
• Contextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wild — Contextual Earnings-22 offers a standardized benchmark for evaluating speech-to-text systems' custom vocabulary recognition in realistic settings.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.