TMF-RSE: Tri-Modal Fusion with Regional Semantics and Evidential Uncertainty for Lung Severity Scoring

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Science & Research — Health & Medical Research, Mathematics & Computational Sciences · Depth: Expert, long

Summary

The TMF-RSE (Tri-Modal Fusion with Regional Semantics and Evidential Uncertainty) framework is a deep learning approach for quantifying lung disease severity from chest imaging. It integrates appearance features from two-dimensional chest inputs, structural features from lung segmentation masks, and semantic features from vision-language models (VLMs). The model employs complementary fusion mechanisms, including semantic gating and structural prior modulation, along with hierarchical interactions across modalities. TMF-RSE also uses evidential regression to provide both severity predictions and uncertainty estimates. Experiments on the Per-COVID-19 CT and RALO datasets demonstrate its superior performance over transformer-based baselines, achieving a Mean Absolute Error (MAE) of 4.02 and Pearson correlation of 0.9629 on Per-COVID-19 validation, and 0.339 MAE / 0.973 PC on RALO geographic extent.

Key takeaway

For AI Scientists and Machine Learning Engineers developing medical imaging solutions, TMF-RSE demonstrates that integrating multi-modal data (appearance, structural, semantic) with explicit uncertainty quantification significantly enhances lung severity assessment. You should consider adopting tri-modal fusion and evidential regression to improve prediction accuracy and provide crucial confidence estimates for clinical decision-making, especially when dealing with complex medical data.

Key insights

Tri-modal fusion with regional semantics and evidential uncertainty improves lung severity quantification.

Principles

• Combine appearance, structural, and semantic features for robust analysis.
• Use evidential regression to provide uncertainty estimates with predictions.
• Employ hierarchical fusion to capture complex cross-modal interactions.

Method

TMF-RSE extracts features via DINOv3-ViT (image), CNN (mask), and LLaVA-Med (VLM with regional prompts). These are fused using semantic gating, structural prior modulation, and hierarchical fusion, then passed to an evidential regression head.

In practice

• Apply SAM3 for efficient lung segmentation mask generation.
• Utilize frozen VLMs with region-specific prompts for semantic encoding.
• Implement uncertainty-based sample removal to identify difficult cases.

Topics

• Lung Severity Scoring
• Multi-modal Fusion
• Evidential Regression
• Vision-Language Models
• Chest Imaging
• Uncertainty Quantification

Best for: Computer Vision Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Dual-Modal Lung Cancer AI: Interpretable Radiology and Microscopy with Clinical Risk Integration — Integrating CT radiology and H&E microscopy with clinical data significantly enhances lung cancer diagnosis and interpretability.
• TRIAGE: Dialectical Reasoning for Explainable Risk Prediction on Irregularly Sampled Medical Time Series with LLMs — TRIAGE uses dialectical LLM reasoning to provide calibrated, continuous risk scores and interpretable rationales for medical time series.
• When Does Quality-Aware Multimodal Fusion Matter? A Leakage-Safe Diagnostic for Decision-Level Dependence — A diagnostic tests if multimodal fusion systems truly utilize modality reliability scores for decision-making.
• Look-Closer-Then-Diagnose: Confidence-Aware Ultrasound VQA via Active Zooming — A novel VLM framework integrates lesion-focused reasoning and uncertainty-aware rewards for improved ultrasound diagnosis.
• Semantic-Anchored Evidential Fusion for Domain-Robust Whole-Slide Survival Analysis — High-level pathology semantics provide domain-invariant representations crucial for robust whole-slide image analysis in cancer prognosis.
• Trusted Multi-View Deep Learning Classification of Fetal Congenital Heart Disease with Feature-level and Decision-level Fusion — The framework integrates multi-view echocardiographic data with uncertainty handling for robust fetal CHD classification.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.