Speech Disfluencies and LLM Confidence: Length Bias and Pragmatic Insensitivity in Brazilian Portuguese

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A pilot study investigated how speech disfluencies, such as filled pauses and repetitions, influence the confidence of the Llama-3.1-8B-Instruct Large Language Model in Brazilian Portuguese. Researchers used 344 turns from the Roda Viva corpus, comparing faithful Conversation Analysis transcriptions with sanitized versions. The study found that the model's confidence is overwhelmingly driven by turn length (${\beta_{\text{std}}} = +14.47, p 0.001$), rather than by pragmatic markers of uncertainty (${\beta_{\text{oral}}} = -3.09, {\beta_{\text{hedges}}} = -0.97$, both non-significant). While disfluency markers showed a human-expected directional effect after controlling for length, this was dwarfed by the length bias. This surface-feature dominance explains the model's "pragmatic blindness" and the substantial divergence observed via ECE (41.95) and OE (4.29) between the two transcription conditions.

Key takeaway

For NLP Engineers developing LLMs for spontaneous speech applications, recognize that your model's confidence may be heavily biased by turn length rather than actual pragmatic uncertainty. If you are evaluating LLM reliability on oral language, consider explicitly controlling for surface features like length. This finding suggests a need to fine-tune models on disfluency-rich corpora to improve their pragmatic sensitivity and ensure more accurate confidence alignment in real-world conversational AI systems.

Key insights

LLM confidence in spontaneous speech is primarily driven by turn length, not pragmatic uncertainty markers.

Principles

• LLMs trained on curated text may struggle with spontaneous oral language features.
• Surface features like length can overshadow pragmatic cues in LLM confidence assessment.

Method

The study contrasted faithful Conversation Analysis transcriptions with sanitized versions of spontaneous speech, applying binned divergence metrics, rank correlation, and multivariate regression.

In practice

• Evaluate LLM performance on spontaneous speech using disfluency-rich datasets.
• Account for length bias when interpreting LLM confidence scores in oral contexts.

Topics

• Large Language Models
• Speech Disfluencies
• LLM Confidence
• Pragmatic Insensitivity
• Brazilian Portuguese
• Llama-3.1-8B-Instruct

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Unveiling the Limits of Large Language Models in Inferring Pragmatic Meaning from Non-Verbal Responses — LLMs significantly struggle to infer pragmatic meaning from non-verbal communication, unlike their performance with verbal cues.
• Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translation — LLMs' verbalized confidence in translation outputs can be as reliable as internal signals, despite no correlation.
• Linguistic Symptoms: Augmented Generation by Retrieval and Reasoning in LLMs under Portuguese-English Variation in Medical Contexts — Linguistic variation significantly impacts smaller LLMs in medical contexts, with larger models showing greater cross-language robustness.
• SICI: A Semantic-Pragmatic Complexity Index Reveals Regime Shifts in LLM Stance Detection — LLM stance detection errors exhibit predictable regime shifts tied to semantic-pragmatic complexity, leading to distinct error patterns.
• LLMs believe false statements even after explicit warnings that they're false — LLMs prioritize statistical patterns in training data over explicit negation, leading to "belief implantation."
• Research on LLM alignment as latent discourse-level regimes vs. token-level filtering? — LLM alignment may stem from global latent space geometry and discourse-level regimes, not just token-level filtering.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.