PS4: Proxy-Supervised Joint Training for Real Target Speaker Extraction

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Speech Processing · Depth: Expert, long

Summary

PS4 is a novel proxy-supervised training framework designed for Target Speaker Extraction (TSE) in real conversational mixtures, addressing the challenge of unavailable large-scale training corpora and clean target speech. The framework introduces two main contributions: first, a large-scale corpus named REAL-PS4, comprising 71,771 training samples derived from four public datasets (AISHELL-4, AliMeeting, AMI, CHiME-6) covering both Chinese and English scenarios. Each sample includes an overlapping speech mixture, per-speaker enrollment audio, a ground-truth transcript, and frame-level voice activity labels. Second, PS4 proposes a proxy-supervised joint training strategy that fine-tunes a BSRNN-based TSE model using four complementary differentiable objectives: ASR cross-entropy, speaker similarity, frame-level voice activity detection, and perceptual audio quality (DNSMOS). This approach, starting from a pre-trained checkpoint, updates only the BSRNN separator. On the REAL-T challenge leaderboard, PS4 achieved 2nd overall rank, securing the best speaker similarity and timing F1 among all submitted systems.

Key takeaway

For Machine Learning Engineers developing Target Speaker Extraction (TSE) systems for real-world conversational audio, consider adopting proxy-supervised training. Your models can achieve robust performance without clean reference speech by leveraging a multi-objective approach combining ASR, speaker similarity, VAD, and perceptual quality losses. This method, demonstrated by PS4's 2nd place on the REAL-T challenge, offers a practical solution to bridge the gap between simulated benchmarks and actual deployment, significantly improving speaker identity preservation and timing accuracy.

Key insights

Proxy-supervised joint training effectively addresses the lack of clean reference speech for Target Speaker Extraction in real-world conversational data.

Principles

• Real-world TSE requires diverse proxy supervision.
• Joint optimization improves linguistic and speaker fidelity.
• Fine-tuning separators on pre-trained models is efficient.

Method

PS4 constructs REAL-PS4 corpus, then fine-tunes a BSRNN separator using a joint optimization of ASR cross-entropy, speaker similarity, VAD, and DNSMOS perceptual quality objectives.

In practice

• Construct training corpora from public meeting datasets.
• Use Whisper large-v3 for linguistic supervision.
• Employ DNSMOS for perceptual quality optimization.

Topics

• Target Speaker Extraction
• Proxy Supervision
• BSRNN
• REAL-T Benchmark
• Multi-objective Optimization

Code references

• TaurenMountain/PS4
• microsoft/dns-challenge

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• PS4: Proxy-Supervised Joint Training for Real Target Speaker Extraction — Proxy-supervised joint training with multiple objectives enables effective Target Speaker Extraction from real conversational mixtures despite limited supervised data.
• Breaking the Quality--Intelligibility Trade-off in Streaming Target Speaker Extraction via Deep-Feature-Anchored Preference Optimization — The quality-intelligibility trade-off in streaming TSE models arises from optimization anchor choice, not architectural constraints.
• Multilingual Multi-Speaker Unit Vocoders: A Systematic Analysis of Discrete Speech Representations — Unit vocoder performance in multilingual settings hinges on cluster size for intelligibility and explicit speaker conditioning for identity preservation.
• Dialectmind@DravidianLang Tech 2026: Zero-Shot Dialectal Tamil Automatic Speech Recognition Using a Large Pretrained Conformer Model — Large pretrained Conformer models enable successful zero-shot dialectal ASR for low-resource languages like Tamil.
• Semi-Supervised Speech Confidence Detection using Pseudo-Labelling and Whisper Embeddings — A novel framework detects speaker confidence by fusing human-engineered features and Whisper embeddings, enhanced by pseudo-labelling for data scarcity.
• ASTRA: A Scalable Next-Generation ATCO Training Simulator with Autonomous Simpilots — Localized speech models and hybrid AI are critical for scalable, objective ATCO training simulation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.