Token-Based Dual-view Fusion and Adaptation of Large Vision Models for Breast Cancer Classification

2026-07-07 · Source: Takara TLDR - Daily AI Papers · Field: Health & Wellbeing — Health & Medical Research, Medical Devices & Health Technology · Depth: Expert, medium

Summary

A novel token-centric dual-view learning framework is introduced for accurate breast cancer classification from mammography, integrating complementary craniocaudal (CC) and mediolateral oblique (MLO) views. This framework unifies prompt-based adaptation and cross-view fusion within a frozen vision transformer backbone, addressing limitations of existing multi-view approaches. It reformulates inter-view interaction using dedicated "fusion tokens" that explicitly encode bidirectional information exchange via cross-attention, acting as intermediate carriers of cross-view dependencies. Unlike conventional single-layer fusion, these fusion modules are inserted at multiple transformer depths, enabling progressive and repeated interaction across the encoder hierarchy. Experiments on VinDr-Mammo and CMMD datasets show consistent improvements, achieving a 50.40% F1-score and 0.8090 AUC on VinDr-Mammo BI-RADS classification, including a 0.10 AUC improvement over a dual-view fusion baseline in the binary setting.

Key takeaway

For Computer Vision Engineers developing medical imaging AI, particularly for breast cancer classification, you should consider implementing token-based dual-view fusion. This approach, which uses dedicated fusion tokens for progressive, multi-depth interaction between mammogram views, significantly improves classification accuracy. You can achieve superior performance, as demonstrated by a 0.10 AUC improvement, by moving beyond single-stage feature fusion and leveraging structured token-level communication within your vision transformer backbones.

Key insights

Token-based fusion and multi-depth interaction in vision transformers enhance breast cancer classification from dual-view mammograms.

Principles

• Multi-depth fusion improves cross-view interaction.
• Fusion tokens enable explicit bidirectional information exchange.
• Preserving view-specific structure is crucial during fusion.

Method

A token-centric framework unifies prompt-based adaptation and cross-view fusion in a frozen vision transformer. Fusion tokens facilitate bidirectional interaction across multiple transformer depths, then reintegrate for hierarchical propagation.

In practice

• Apply fusion tokens for structured cross-modal interaction.
• Implement multi-depth fusion in transformer architectures.
• Use prompt-based adaptation with frozen vision backbones.

Topics

• Breast Cancer Classification
• Mammography
• Vision Transformers
• Multi-view Learning
• Token-based Fusion
• Prompt-based Adaptation

Code references

• dpetrini/multiple-view

Best for: AI Scientist, Computer Vision Engineer, Research Scientist

Related on AIssential

• Token-Based Dual-view Fusion and Adaptation of Large Vision Models for Breast Cancer Classification — A token-centric, multi-depth fusion framework enhances breast cancer classification by integrating complementary mammography views.
• Dual-Modal Lung Cancer AI: Interpretable Radiology and Microscopy with Clinical Risk Integration — Integrating CT radiology and H&E microscopy with clinical data significantly enhances lung cancer diagnosis and interpretability.
• Token-Sparse Medical Multimodal Reasoning via Dual-Stream Reinforcement Learning — ViToS employs dual-stream reinforcement learning and active visual token pruning to significantly improve medical multimodal reasoning with sparse visual evidence.
• Multimodal Fusion for Fine-Grained Classification of Breast Fibroadenoma and Phyllodes Tumors — Multimodal fusion of imaging, clinical, and text data significantly improves fine-grained breast lesion classification accuracy.
• Evidential learning driven Breast Tumor Segmentation with Stage-divided Vision-Language Interaction — TextBCS enhances breast tumor segmentation in MRI by combining text prompts with evidential learning to address low contrast and blurred boundaries.
• Trusted Multi-View Deep Learning Classification of Fetal Congenital Heart Disease with Feature-level and Decision-level Fusion — The framework integrates multi-view echocardiographic data with uncertainty handling for robust fetal CHD classification.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.