Multimodal Fusion for Fine-Grained Classification of Breast Fibroadenoma and Phyllodes Tumors

2026-07-02 · Source: Computer Vision and Pattern Recognition · Field: Health & Wellbeing — Artificial Intelligence & Machine Learning, Health & Medical Research, Medical Specialties & Subspecialties · Depth: Expert, quick

Summary

A new clinically guided multimodal framework addresses the challenge of distinguishing breast fibroadenoma (FA) and phyllodes tumor (PT) due to their overlapping appearances on B-mode ultrasound. Researchers constructed the FAPT-M Dataset, comprising 910 patients with strictly reviewed ultrasound images, structured clinical attributes, and ultrasound diagnostic descriptions. The proposed framework integrates DenseNet-based visual encoding, CLIP-inspired text encoding, and lightweight clinical encoding. It further introduces clinical-conditioned adaptive modulation, cross-modal Transformer fusion, and dual-path representation learning to improve feature alignment and multimodal interaction. Under patient-level five-fold cross-validation, the method achieved an accuracy of 77.64%, an F1-score of 73.38%, and an AUC of 89.74%, outperforming representative CNN-, Transformer-, and vision-language-based baselines.

Key takeaway

For AI Scientists and Machine Learning Engineers working on medical image classification, this research demonstrates that integrating multimodal data, specifically ultrasound images, clinical attributes, and diagnostic descriptions, can significantly improve the accuracy of fine-grained distinctions like breast fibroadenoma and phyllodes tumors. You should consider developing similar clinically guided multimodal frameworks for other challenging diagnostic tasks to enhance preoperative decision-making and reduce misclassification rates.

Key insights

Multimodal fusion of imaging, clinical, and text data significantly improves fine-grained breast lesion classification accuracy.

Principles

• Complementary data modalities enhance diagnostic accuracy.
• Clinical context can modulate visual and textual features.
• Dual-path learning improves cross-modal feature alignment.

Method

Integrates DenseNet visual, CLIP text, and lightweight clinical encoding, using clinical-conditioned adaptive modulation, cross-modal Transformer fusion, and dual-path representation learning for feature alignment.

In practice

• Combine ultrasound with clinical and text data.
• Employ Transformer fusion for multimodal interaction.
• Utilize dual-path learning for feature alignment.

Topics

• Multimodal Fusion
• Breast Lesion Classification
• Deep Learning
• Ultrasound Imaging
• Fibroadenoma
• Phyllodes Tumor
• Transformer Networks

Best for: Computer Vision Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Dual-Modal Lung Cancer AI: Interpretable Radiology and Microscopy with Clinical Risk Integration — Integrating CT radiology and H&E histopathology via dual-modal AI improves lung cancer diagnosis and interpretability.
• MLFFM-SegDiff: A Multi-Level Feature Fusion Diffusion Model for Skin Lesion Segmentation — Diffusion models can be enhanced for medical image segmentation by improving cross-level feature interaction and boundary detail recovery.
• Systematically decoding pathological morphologies and molecular profiles with unified multimodal embedding — Multi-Embed unifies morphological and molecular data for superior disease understanding and prediction.
• Multi-FRuGaL: Multimodal Flexible Redundancy-aware Decomposed Gated Learning for Cancer Diagnosis and Prognosis — Adaptive, redundancy-aware gated fusion improves cancer prognosis with incomplete multimodal patient data.
• Clinically-Informed Modeling for Pediatric Brain Tumor Classification from Whole-Slide Histopathology Images — Contrastive learning, especially with expert-guided hard negatives, improves fine-grained pediatric brain tumor classification from WSIs.
• Trusted Multi-View Deep Learning Classification of Fetal Congenital Heart Disease with Feature-level and Decision-level Fusion — The framework integrates multi-view echocardiographic data with uncertainty handling for robust fetal CHD classification.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.