Audio-Text Cross-Attention with Psycholinguistic Support Features for Ambivalence/Hesitancy Recognition

2026-07-15 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

An audio-text system designed for the 11th ABAW Competition's Ambivalence/Hesitancy Video Recognition Challenge processes videos by excluding visual frames and representing them as overlapping 5-second windows, aligned with transcript timestamps. Each window integrates a 320-dimensional prosodic audio descriptor, a 768-dimensional emotion-oriented RoBERTa embedding, and 74 handcrafted features specifically capturing uncertainty, hedging, and attitudinal conflict. Audio and text components are fused using temporal cross-attention, while the support features are injected before gated multiple-instance learning (MIL) pooling to modulate window importance. The system averages predictions from five independently initialized models, achieving an an average precision of 0.875 and a macro-F1 of 0.72 on the labeled public development set. The source code is publicly available.

Key takeaway

For NLP Engineers developing emotion recognition systems, consider integrating psycholinguistic support features alongside audio and text modalities. Your models can achieve higher precision and F1 scores by using temporal cross-attention for fusion and gated multiple-instance learning for feature modulation. Explore the publicly available source code to adapt this multi-modal, ensemble approach for similar complex emotional state detection tasks.

Key insights

Combining audio, text, and psycholinguistic features via cross-attention improves ambivalence recognition.

Principles

• Exclude visual frames for specific emotional cues.
• Integrate diverse feature types for robust recognition.
• Ensemble multiple models for performance stability.

Method

The system processes 5-second audio-text windows, fusing prosodic audio and RoBERTa text via temporal cross-attention, then injecting 74 handcrafted psycholinguistic features before gated MIL pooling.

In practice

• Use 5-second overlapping windows for temporal context.
• Apply RoBERTa for emotion-oriented text embeddings.
• Implement gated MIL pooling for feature modulation.

Topics

• Ambivalence Recognition
• Hesitancy Detection
• Audio-Text Fusion
• Cross-Attention Networks
• Psycholinguistic Features
• RoBERTa Embeddings
• Multiple-Instance Learning

Code references

• Liga-de-IA-PUCPR/abaw-11-ah-challenge

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• A Calibrated Multimodal Ensemble for Ambivalence/Hesitancy Recognition: System Description and Private-Test Submission Strategy — A calibrated multimodal ensemble system accurately recognizes ambivalence and hesitancy from video, demonstrating strong generalization to unseen participants.
• McMaster NLP at SemEval-2026 Task 2: A Lightweight Multi-Feature System for Predicting Emotional Valence and Arousal over Time — A lightweight, multi-feature system predicts emotional valence and arousal from longitudinal language data using diverse signals.
• Beyond Parallel Tracking: Interactive Multi-Feature Fusion Drives Semantic Reconstruction from Non-invasive Brain Recordings — Non-invasive brain-to-text decoding improves significantly by interactively fusing static lexical and dynamic contextual features.
• Multi-View Attention Multiple-Instance Learning Enhanced by LLM Reasoning for Cognitive Distortion Detection — Combining LLMs with MIL and ELB components improves cognitive distortion detection by enhancing interpretability and reasoning.
• Cross-Lingual Sentiment Misalignment: Auditing Multilingual Language Models for Inversion Risk, Dialectal Representation, and Affective Stability — Multilingual models exhibit significant cross-lingual sentiment misalignment, inversion, and dialectal bias, especially for low-resource languages like Bengali.
• SCoPE: Shift-Aware Speaker-Conditioned Priors for Emotion Recognition in Conversations — SCoPE dynamically fuses speaker-conditioned emotional history with multimodal evidence, guided by predicted emotion shifts, for robust ERC.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.