Token-Based Dual-view Fusion and Adaptation of Large Vision Models for Breast Cancer Classification

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Medical Devices & Health Technology, Health & Medical Research · Depth: Expert, extended

Summary

A novel token-centric dual-view learning framework is proposed for accurate breast cancer classification using mammography. This framework unifies prompt-based adaptation and cross-view fusion within a frozen MedSigLIP vision transformer backbone. It reformulates inter-view interaction as structured token-level communication, where dedicated fusion tokens explicitly encode bidirectional information exchange between craniocaudal (CC) and mediolateral oblique (MLO) views via cross-attention. Unlike conventional methods, fusion modules are inserted at multiple transformer depths, enabling progressive and repeated interaction across the encoder hierarchy. Experiments on VinDr-Mammo and CMMD datasets demonstrate consistent improvements, achieving 50.40% F1-score and 0.8090 AUC on VinDr-Mammo BI-RADS classification, and a 0.10 AUC improvement over a dual-view fusion baseline in the binary setting.

Key takeaway

For Machine Learning Engineers adapting large vision models for multi-view medical imaging, this framework provides a robust method to integrate complementary information from different views. By implementing multi-depth token-based fusion and shared prompt learning, you can achieve superior performance and parameter efficiency compared to conventional feature-level fusion. Consider optimizing prompt depth and fusion block placement to maximize classification accuracy on your specific datasets.

Key insights

Token-based fusion and multi-depth interaction effectively integrate complementary mammography views within frozen vision transformers.

Principles

• Multi-depth cross-view interaction improves hierarchical information integration.
• Dedicated fusion tokens preserve cross-view dependencies.
• Moderate, well-spaced fusion blocks provide optimal integration.

Method

A two-stage framework first applies deep shared-view prompt learning, then introduces token-based cross-view fusion via bidirectional cross-attention and fusion token insertion at multiple transformer depths, concatenating final embeddings for classification.

In practice

• Utilize MedSigLIP as a frozen vision encoder for medical imaging.
• Implement token-based cross-attention for multi-view data fusion.
• Optimize prompt depth and fusion block placement for specific datasets.

Topics

• Breast Cancer Classification
• Mammography
• Multi-view Fusion
• Vision Transformers
• Prompt Learning
• Cross-Attention

Code references

• PartAI-Projects/CrossViewTokenFusion

Best for: Computer Vision Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Token-Based Dual-view Fusion and Adaptation of Large Vision Models for Breast Cancer Classification — A token-centric, multi-depth fusion framework enhances breast cancer classification by integrating complementary mammography views.
• Layer-Specific Prompt Fusion Discovery via Differentiable Search in Vision Foundation Models — Hybrid prompt fusion, discovered via differentiable search, significantly improves visual prompt tuning in ViTs.
• SegMoTE: Token-Level Mixture of Experts for Medical Image Segmentation — SegMoTE adapts SAM for medical image segmentation using token-level experts and progressive prompting, achieving high performance with minimal data.
• Late-Layer Fusion is Enough: Dual-Path Vision Token Routing for Multimodal Large Language Models under Visual Saturation — Vision tokens in MLLMs saturate early, making deep symmetric processing inefficient; late-layer fusion is sufficient.
• Token-Based Dual-view Fusion and Adaptation of Large Vision Models for Breast Cancer Classification — Token-based fusion and multi-depth interaction in vision transformers enhance breast cancer classification from dual-view mammograms.
• IEEE Transactions on Neural Networks and Learning Systems, Volume 37, Issue 5, May 2026 — The volume showcases diverse advancements in neural networks, from foundational surveys to specialized applications and novel architectures.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.