Token-Based Dual-view Fusion and Adaptation of Large Vision Models for Breast Cancer Classification

2026-07-07 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Health & Medical Research · Depth: Expert, quick

Summary

A token-centric dual-view learning framework is proposed for accurate breast cancer classification from mammography, integrating craniocaudal (CC) and mediolateral oblique (MLO) views. This framework unifies prompt-based adaptation and cross-view fusion within a frozen vision transformer backbone. It reformulates inter-view interaction as structured token-level communication, where dedicated fusion tokens explicitly encode bidirectional information exchange via cross-attention. Fusion modules are inserted at multiple transformer depths, enabling progressive interaction and hierarchical propagation of complementary information. Experiments on VinDr-Mammo and CMMD datasets show consistent improvements over baselines. On VinDr-Mammo BI-RADS classification, it achieved a 50.40% F1-score and 0.8090 AUC, including a 0.10 AUC improvement over a dual-view fusion baseline in the binary setting.

Key takeaway

For Machine Learning Engineers developing multi-view medical image analysis models, this token-centric fusion approach offers a robust method to integrate complementary information. You should consider implementing multi-depth token-level communication via cross-attention within your vision transformer backbones to improve classification accuracy, especially when dealing with complex diagnostic tasks like breast cancer detection from mammograms. This can lead to more effective hierarchical information propagation and better diagnostic performance.

Key insights

A token-centric, multi-depth fusion framework enhances breast cancer classification by integrating complementary mammography views.

Principles

• Reformulate inter-view interaction as structured token-level communication.
• Enable progressive, repeated interaction across the encoder hierarchy.

Method

Dedicated fusion tokens explicitly encode bidirectional information exchange between views via cross-attention, inserted at multiple transformer depths, then reintegrated and refined by subsequent layers.

In practice

• Apply token-based fusion for integrating complementary medical imaging views.
• Utilize multi-depth interaction to propagate hierarchical information effectively.

Topics

• Breast Cancer Classification
• Mammography
• Vision Transformers
• Multi-view Learning
• Token-based Fusion
• Medical Imaging

Best for: Computer Vision Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Dual-Modal Lung Cancer AI: Interpretable Radiology and Microscopy with Clinical Risk Integration — Integrating CT radiology and H&E microscopy with clinical data significantly enhances lung cancer diagnosis and interpretability.
• OTCHA: Optimal Transport-driven Confidence-aware Latent Hub Alignment for Multi-View Medical Image Classification — OTCHA refines multi-view medical image tokens using confidence-aware optimal transport and shared latent hubs to prevent fusion contamination.
• Meta-encoder: a unified integration framework for multiple pathological foundation models in cancer detection — The Meta-Encoder integrates multiple pathological foundation models to improve cancer detection and molecular characterization.
• Evidential learning driven Breast Tumor Segmentation with Stage-divided Vision-Language Interaction — TextBCS enhances breast tumor segmentation in MRI by combining text prompts with evidential learning to address low contrast and blurred boundaries.
• Trusted Multi-View Deep Learning Classification of Fetal Congenital Heart Disease with Feature-level and Decision-level Fusion — The framework integrates multi-view echocardiographic data with uncertainty handling for robust fetal CHD classification.
• BreastGPT: A Multimodal Large Language Model for the Full Spectrum of Breast Cancer Clinical Routine — Workflow-aligned data and cross-scale visual modeling are critical for clinically grounded medical MLLMs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.