Transformer-based segmentation of prosodic boundaries in Brazilian Portuguese

2026-07-09 · Source: cs.CL updates on arXiv.org · Depth: Unknown, long

Summary

SAMPA, a novel Whisper-based segmenter, automatically identifies terminal prosodic boundaries in Brazilian Portuguese (BP) speech by fine-tuning Whisper large-v3. This approach reformulates prosodic segmentation as a sequence-to-sequence transcription task, where the model outputs transcribed text with explicit boundary markers. Researchers fine-tuned Whisper large-v3 on manually segmented recordings from the NURC-SP dataset, evaluating various training and test-time filtering configurations, including out-of-distribution testing on the MuPe-Diversidades dataset. SAMPA achieved competitive boundary-detection performance, reaching an F₁=0.731 on the held-out NURC-SP test split and F₁=0.796 on MuPe-Diversidades. N-gram and acoustic-visual analyses revealed that the model leverages morphosyntactic, semantic, and prosodic cues for boundary detection, with high-pass filtering configurations showing better generalization to cleaner, more geographically diverse out-of-distribution data.

Key takeaway

For speech technology developers building robust Brazilian Portuguese applications, SAMPA demonstrates a highly effective method for prosodic boundary segmentation. You should consider fine-tuning large pre-trained ASR models like Whisper large-v3, integrating explicit boundary markers into the transcription task. This approach yields strong performance, achieving F₁ scores up to 0.796 on diverse datasets. Furthermore, strategically applying high-pass audio filtering during training can enhance your model's generalization to cleaner, out-of-distribution speech.

Key insights

Prosodic segmentation for Brazilian Portuguese can be effectively achieved by fine-tuning Whisper large-v3 to embed boundary markers in transcriptions.

Principles

• Language-specific intonation patterns necessitate tailored segmentation strategies.
• Reformulating segmentation as a transcription task allows Transformer models to learn acoustic and linguistic cues.
• Pre-trained models like Whisper offer robustness and multilingual competence for adaptation.

Method

Fine-tune Whisper large-v3 by adding a special delimiter token ("!!!!!") to its vocabulary, training it to insert this token for terminal prosodic boundaries within transcribed speech. Preprocessing involves concatenating adjacent speech segments under 30 seconds, maintaining speaker continuity, and filtering noisy audio.

In practice

• Extend pre-trained ASR models like Whisper for new sequence-to-sequence tasks by incorporating custom delimiter tokens.
• Apply high-pass audio filtering during training to improve model generalization to out-of-distribution speech data.
• Concatenate short, speaker-continuous audio segments to optimize input for Transformer models.

Topics

• Prosodic Segmentation
• Brazilian Portuguese
• Whisper large-v3
• Transformer Models
• Speech Recognition
• Audio Filtering

Related on AIssential

• Transformer-based segmentation of prosodic boundaries in Brazilian Portuguese — SAMPA, a Whisper-based segmenter, effectively identifies prosodic boundaries in Brazilian Portuguese speech using deep learning.
• Software for Automatic Speech Recognition via Whisper models applied to Oral History interviews in the Portuguese language — Ethos AT provides private, accessible automatic speech transcription for Portuguese using Whisper models.
• Robustness and Diversity Evaluation on ProsSegue-ML: a Free Prosodic Segmentation Tool for Brazilian Portuguese — ProsSegue-ML's robustness and bias are evaluated for Brazilian Portuguese prosodic segmentation.
• Speech-Guided Multimodal Learning for Vocal Tract Segmentation in Real-Time MRI — Multimodal supervision during training can enhance single-modality inference for challenging medical image segmentation tasks.
• A Comparative Study of Pretrained Transformer Models for Quranic ASR: Speech Representations, Label Formats, and Dataset Composition — Fine-tuning pretrained Transformers with specific speech representations and label formats significantly improves Quranic ASR accuracy and efficiency.
• Using embeddings to predict spoken word duration and pitch in Mandarin monosyllabic words — Contextualized embeddings effectively predict both duration and pitch contours for Mandarin monosyllabic words.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.