How Good LLMs Are at Answering Bangla Medical Visual Questions? Dataset and Benchmarking

2026-05-18 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Medical Devices & Health Technology · Depth: Expert, quick

Summary

A new dataset, BanglaMedVQA, has been introduced to benchmark Large Language Models (LLMs) and Large Vision Language Models (LVLMs) on medical visual question answering (MedVQA) in Bangla. This dataset consists of clinically validated image-question-answer pairs, addressing a significant gap for one of the world's most widely spoken languages. Initial evaluations of current foundation models, including Gemini and GPT-4.1 mini, on BanglaMedVQA reveal substantially lower performance compared to English MedVQA benchmarks. Models struggle with specialized diagnostic questions and fine-grained medical reasoning, indicating severe limitations in low-resource language contexts. While some open-source models like Gemma-3 occasionally show better general performance, they also fail on clinically complex questions, highlighting the need for improved evaluation methods and model capabilities.

Key takeaway

For AI Scientists and Machine Learning Engineers developing medical AI, the BanglaMedVQA dataset highlights critical performance gaps in low-resource languages. Your current foundation models, even top-tier ones like Gemini and GPT-4.1 mini, are likely insufficient for accurate clinical reasoning in Bangla. Prioritize research into language-specific fine-tuning and advanced reasoning architectures to address these limitations and ensure equitable access to medical AI.

Key insights

BanglaMedVQA dataset reveals current LLMs and LVLMs perform poorly on medical visual questions in Bangla.

Principles

• Low-resource languages pose significant challenges for MedVQA.
• Current models lack fine-grained medical reasoning capabilities.

In practice

• Develop specialized models for low-resource medical VQA.
• Focus on improving fine-grained medical reasoning.

Topics

• BanglaMedVQA Dataset
• Medical Visual Question Answering
• Large Vision Language Models
• Low-Resource Language Performance
• Clinical Reasoning

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Multilingual Hematology Visual Question Answering Dataset — Multilingual VQA datasets are crucial for equitable medical AI, bridging language gaps in healthcare.
• Active Evidence-Seeking and Diagnostic Reasoning in Large Language Models for Clinical Decision Support — LLMs exhibit a substantial performance gap in active, multi-turn clinical diagnosis compared to static, full-context evaluations.
• OpenMedQ: Broad Open Pretraining for Medical Vision-Language Models — Broad open pretraining on diverse medical data enables SOTA performance with smaller models.
• BreastGPT: A Multimodal Large Language Model for the Full Spectrum of Breast Cancer Clinical Routine — Workflow-aligned data and cross-scale visual modeling are critical for clinically grounded medical MLLMs.
• Benchmarking GPT Models for Conversational AI Systems: Can AI Read a Doctor’s Notes? — LLMs struggle with medical abbreviation expansion, especially context-dependent ones, despite larger models performing better.
• Scalable Training of Spatially Grounded 2D Vision-Language Models for Radiology — Spatially grounded radiology VLMs can be trained at scale without manual annotations using LLM-based curation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.