When Fine-Tuning Changes the Evidence: Architecture-Dependent Semantic Drift in Chest X-Ray Explanations

2026-04-09 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

A study on medical image classification reveals that fine-tuning deep learning models can lead to "semantic drift," where the visual evidence supporting predictions changes despite stable diagnostic performance. Researchers evaluated DenseNet201, ResNet50V2, and InceptionV3 on a five-class chest X-ray task using a two-stage training protocol. They quantified drift using reference-free metrics for spatial localization and structural consistency of attribution maps. While coarse anatomical localization remained stable across architectures, overlap IoU showed significant architecture-dependent reorganization of evidential structure. The stability rankings of explanation methods like LayerCAM and GradCAM++ also reversed, indicating that explanation stability is an interaction between architecture, optimization phase, and the attribution objective.

Key takeaway

For research scientists developing medical image classification models, you should explicitly evaluate the stability of visual explanations after fine-tuning. Do not assume that stable classification performance implies stable underlying visual reasoning. Incorporate metrics like overlap IoU and compare different attribution methods (e.g., LayerCAM, GradCAM++) to detect architecture-dependent semantic drift, ensuring your model's interpretability remains consistent and reliable.

Key insights

Fine-tuning medical image models can alter visual evidence (semantic drift) even with stable diagnostic accuracy.

Principles

• Accuracy does not guarantee explanation stability.
• Semantic drift is architecture-dependent.
• Explanation stability is multi-factorial.

Method

Evaluated DenseNet201, ResNet50V2, InceptionV3 on a five-class chest X-ray task using a two-stage training protocol and quantified semantic drift with reference-free metrics.

In practice

• Assess explanation stability post-fine-tuning.
• Compare LayerCAM and GradCAM++ for stability.
• Consider architecture's impact on visual evidence.

Topics

• Transfer Learning
• Fine-Tuning
• Semantic Drift
• Chest X-ray Explanations
• Attribution Maps

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Tracking Representation Dynamics in Large Language Models with Persistent Homology — Persistent homology uncovers internal representation dynamics in LLMs during fine-tuning, showing early topological reorganization and objective-specific trajectories.
• HiSem: Hierarchical Semantic Disentangling for Remote Sensing Image Change Captioning — Disentangling semantic granularities hierarchically improves remote sensing image change captioning performance.
• Quantifying Explanation Consistency: The C-Score Metric for CAM-Based Explainability in Medical Image Classification — The C-Score quantifies explanation consistency in CAM methods, revealing model instability beyond classification metrics.
• When Probing Accuracy Saturates, Fragility Resolves: A Complementary Metric for LLM Pre-Training Analysis — Fragility, a new metric, reveals LLM pre-training dynamics invisible to standard probing accuracy.
• Fine-Tuning Gemma 4 for Vision — Fine-tuning Gemma 4 E2B for medical VQA improves output structure but faces challenges with complex medical reasoning.
• LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data — LLMs struggle with epistemic self-awareness on structured clinical data, but cross-model attribution divergence can detect blind spots and improve reliability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.