PS4: Proxy-Supervised Joint Training for Real Target Speaker Extraction

2026-07-09 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Speech & Audio Processing · Depth: Expert, quick

Summary

PS4 is a novel proxy-supervised training framework designed for Target Speaker Extraction (TSE) in real conversational mixtures, addressing the scarcity of large-scale supervised training data. The framework's first contribution is the creation of a substantial corpus comprising 71,771 training samples, sourced from four public datasets covering both Chinese and English scenarios. Each sample integrates an overlapping speech mixture, per-speaker enrollment audio, a ground-truth transcript, and frame-level voice activity labels. Second, PS4 introduces a proxy-supervised joint training strategy that fine-tunes a BSRNN-based TSE model. This fine-tuning process utilizes four complementary differentiable objectives: ASR cross-entropy, speaker similarity, frame-level voice activity detection, and perceptual audio quality. Starting from a pre-trained checkpoint, only the BSRNN separator is updated. PS4 achieved 2nd place overall on the REAL-T challenge leaderboard, demonstrating the best speaker similarity and timing F1 among all submissions.

Key takeaway

For AI Scientists developing Target Speaker Extraction models for real conversational environments, PS4 demonstrates a viable path to overcome data scarcity. You should explore proxy-supervised joint training, integrating diverse objectives such as ASR cross-entropy, speaker similarity, frame-level voice activity detection, and perceptual audio quality. This strategy, particularly when fine-tuning only the separator of a pre-trained BSRNN model, can yield strong performance, as evidenced by PS4's top speaker similarity on the REAL-T challenge.

Key insights

Proxy-supervised joint training with multiple objectives enables effective Target Speaker Extraction from real conversational mixtures despite limited supervised data.

Principles

• Proxy supervision enables real-world TSE.
• Multi-objective training improves model robustness.
• Large synthetic corpora aid data scarcity.

Method

Fine-tune a BSRNN-based TSE model using a proxy-supervised joint training strategy. This involves four differentiable objectives: ASR cross-entropy, speaker similarity, frame-level VAD, and perceptual audio quality. Only the separator is updated.

In practice

• Build large corpora from public datasets.
• Combine ASR, similarity, VAD, and quality objectives.
• Fine-tune only the separator in pre-trained models.

Topics

• Target Speaker Extraction
• Proxy Supervision
• Joint Training
• BSRNN
• Speech Processing
• REAL-T Challenge

Best for: NLP Engineer, AI Scientist, Research Scientist

Related on AIssential

• PS4: Proxy-Supervised Joint Training for Real Target Speaker Extraction — Proxy-supervised joint training effectively addresses the lack of clean reference speech for Target Speaker Extraction in real-world conversational data.
• ART: A Faster Way to Train Speech Recognition Models — ART accelerates RNN-T training by combining CTC for baseline alignment with residual learning for refinement.
• DialogueSidon: Recovering Full-Duplex Dialogue Tracks from In-the-Wild Dialogue Audio — DialogueSidon recovers distinct speaker tracks from overlapping, real-world conversations.
• Semi-Supervised Speech Confidence Detection using Pseudo-Labelling and Whisper Embeddings — A novel framework detects speaker confidence by fusing human-engineered features and Whisper embeddings, enhanced by pseudo-labelling for data scarcity.
• VieSpeaker: A Large-Scale Vietnamese Speaker Recognition Dataset Beyond Visual Dependency — Face-independent dataset construction using textual metadata and LLM reasoning enables large-scale speaker recognition resources for under-resourced languages.
• From A to B to A: Palindromic Zero-Shot Voice Conversion with Non-Parallel Data — A voice conversion framework uses KNN over WavLM to create synthetic training pairs from non-parallel data for zero-shot, multilingual VC.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.