Audio Sentiment Analysis via Distillation and Cross-Modal Integration of Generated Multilingual Transcripts

2026-07-09 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

Radu Tudor Ionescu proposes a novel knowledge distillation framework for audio sentiment polarity classification, integrating speech with automatically generated multilingual text. The multimodal teacher model employs a cascaded cross-modal transformer to combine audio, ASR-generated English transcripts, and NMT-translated German, French, and Spanish texts. This teacher's knowledge is then transferred to an efficient audio-only WavLM student model. Experiments on the large-scale MSP-Podcast corpus, comprising 169,190 training, 34,399 development, and 46,294 test-1 samples, demonstrate significant performance gains. The multimodal teacher achieved up to +5.89% macro-F1 and +5.15% accuracy over the unimodal WavLM baseline. Crucially, the distilled audio-only student boosted performance by +1.54% macro-F1 and +0.81% accuracy without introducing computational overhead during inference. Ablation studies confirmed the utility of both automatic transcripts and translations.

Key takeaway

For Machine Learning Engineers developing audio sentiment analysis systems, you can achieve substantial performance improvements for your audio-only models without increasing inference latency. Implement a knowledge distillation pipeline where a multimodal teacher, trained on audio and automatically generated multilingual text transcripts, transfers its insights to a lightweight audio-only student. This approach allows your deployed models to benefit from rich textual context during training, while maintaining real-time efficiency.

Key insights

Multimodal integration of audio with generated multilingual text, combined with knowledge distillation, significantly enhances audio-only sentiment analysis performance without inference overhead.

Principles

• Automatically generated multilingual text provides valuable sentiment polarity clues for multimodal models.
• Knowledge distillation effectively transfers complex multimodal model insights to efficient unimodal models.
• Learning under privileged information enables using extra modalities solely during training for inference-time efficiency.

Method

The method involves generating English transcripts via Faster-Whisper ASR, translating them into German, French, and Spanish via NLLB-200 NMT, encoding modalities with specific backbones, fusing them with a Cascaded Cross-Modal Transformer, and distilling knowledge to an audio-only WavLM student.

In practice

• Generate multilingual text transcripts from audio using ASR and NMT for richer training data.
• Implement cascaded cross-modal transformers for robust multimodal feature integration.
• Distill knowledge from complex multimodal models into efficient unimodal models for deployment.

Topics

• Audio Sentiment Analysis
• Knowledge Distillation
• Cross-Modal Transformers
• Multimodal Learning
• Automatic Speech Recognition
• Neural Machine Translation

Code references

• andreidurdun/cross-modal-audio-sentiment
• SYSTRAN/faster-whisper

Best for: AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Audio Sentiment Analysis via Distillation and Cross-Modal Integration of Generated Multilingual Transcripts — Multimodal integration of audio and generated multilingual text significantly boosts audio sentiment analysis performance.
• Fast Speech Foundation Model Distillation Using Interleaved Stacking — Interleaved stacking accelerates SFM distillation training while preserving performance by maintaining layer position.
• Continuous Audio Thinking for Large Audio Language Models — CoAT enables LALMs to retain and organize acoustic information in a continuous latent space, improving understanding without added inference cost.
• HybridCodec: Fast Dual-Stream, Semantically Enhanced Neural Audio Codec — HybridCodec combines dual-stream and distillation for fast, semantically specialized neural audio coding without SSL inference overhead.
• RABIT: Rationale-Based Distillation Towards Interpretable Automatic Speaking Assessment via a Small Language Model — RABIT distills LLM-generated rationales into a small language model for interpretable automatic speaking assessment.
• AudioX-Turbo: A Unified Framework for Efficient Anything-to-Audio Generation — AudioX-Turbo unifies multimodal audio generation with efficient few-step diffusion via teacher-student distillation and a large dataset.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.