NJUST-KMG at MedGenVidQA 2026: Cascade Multi-modal Alignment with Gaussian Soft Priors for Medical Visual Answer Localization

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Health & Medical Research · Depth: Expert, quick

Summary

The NJUST-KMG system, developed for the Medical Visual Answer Localization (MVAL) task at MedGenVidQA 2026, addresses challenges in precisely locating surgical or instructional steps within medical videos, which often suffer from audio-visual asynchrony and visual homogeneity. This system proposes a Cascade Multi-modal Alignment Framework that integrates Large Language Models (LLMs) to bridge the semantic-temporal gap. Its pipeline employs WhisperX for accurate word-level speech transcription, followed by Gemini3 acting as a high-level semantic ranker to generate multi-scale textual priors. These discrete scores are then converted into a continuous 1D Gaussian Soft Prior, which is injected as an attention bias into the cross-modal fusion network. This mechanism maintains global temporal context while directing the model's focus to query-relevant frames, achieving a competitive IoU@0.7 of 67.5% on the validation leaderboard.

Key takeaway

For Machine Learning Engineers developing medical video analysis systems, you should consider integrating multi-modal alignment techniques to overcome audio-visual asynchrony and visual homogeneity. Employing tools like WhisperX for precise speech transcription and LLMs such as Gemini3 for semantic prior generation can significantly improve localization accuracy. Specifically, applying continuous Gaussian Soft Priors as attention biases can guide your models to focus on critical, query-relevant frames, enhancing performance on strict evaluation metrics.

Key insights

A Cascade Multi-modal Alignment Framework uses LLMs and Gaussian Soft Priors to precisely localize events in medical videos despite asynchrony.

Principles

• LLMs can bridge semantic-temporal gaps.
• Continuous priors enhance cross-modal fusion.
• Speech transcription improves textual anchoring.

Method

The system transcribes speech with WhisperX, generates multi-scale textual priors using Gemini3, converts them to a 1D Gaussian Soft Prior, and injects this as an attention bias into a cross-modal fusion network.

In practice

• Integrate WhisperX for precise speech-to-text.
• Utilize LLMs like Gemini3 for semantic ranking.
• Apply Gaussian priors for temporal context guidance.

Topics

• Medical Visual Answer Localization
• Multi-modal Alignment
• Large Language Models
• Gaussian Soft Priors
• WhisperX
• Gemini3

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Hierarchical Semantic-Constrained Heterogeneous Graph for Audio-Visual Event Localization — HSCHG improves open-vocabulary audio-visual event localization by combining Euclidean graph modeling with hyperbolic space for hierarchical semantic alignment.
• Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain — The Spectral Alignment Score (SAS) offers an asymmetric diagnostic for VLM modality imbalance, revealing directional information asymmetries in medical image-text data.
• AlignAtt4LLM: Fast AlignAtt for Decoder-Only LLMs at IWSLT 2026 Simultaneous Speech Translation Task — Adapting AlignAtt for decoder-only LLMs enables simultaneous speech translation by re-engineering attention mechanisms.
• Noise-Aware Visual Representation Learning for Medical Visual Question Answering — Denoising visual embeddings before LLM projection significantly improves Med-VQA robustness against input noise.
• Temporally-Aligned Evaluation for Audio-Driven Talking Head Generation — Evaluating dynamic generative models requires sequence alignment, not just frame-wise comparison, to handle natural temporal variations.
• Speech-Guided Multimodal Learning for Vocal Tract Segmentation in Real-Time MRI — Multimodal supervision during training can enhance single-modality inference for challenging medical image segmentation tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.