Systematic Performance Degradation in Indic Vision-Language Models: Evidence from Hindi and Telugu

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

HinTel-AlignBench is a new, comprehensive evaluation framework and benchmark designed to assess Hindi and Telugu vision-language models (VLMs) using English-aligned samples. Developed to address shortcomings in existing multilingual VLM benchmarks, which often feature unverified auto-translations, limited task coverage, small sample sizes, and insufficient culturally grounded content, HinTel-AlignBench generates 4k question-answer pairs per language. This is achieved through a semi-automated translation process combined with human verification, spanning five distinct domains. These domains include adapted English datasets like VQAv2, RealWorldQA, and CLEVR-Math, alongside native Indic sets such as JEE for STEM and VAANI for cultural grounding. Evaluations of current open and closed-source VLMs using this benchmark consistently reveal performance degradation when transitioning from English to Indic languages, with average drops of 8.3 points for Hindi and 5.5 points for Telugu across four of five tasks. The framework also identifies key failure modes and establishes reproducible baselines for future multilingual multimodal evaluations.

Key takeaway

For machine learning engineers developing or evaluating vision-language models for Indic languages, recognize that current benchmarks are inadequate and lead to systematic performance degradation. You should prioritize using culturally grounded and human-verified evaluation frameworks, such as HinTel-AlignBench, to accurately assess model capabilities. This approach will help you identify specific failure modes and build more robust, equitable multilingual AI systems, moving beyond English-centric performance metrics.

Key insights

Multilingual VLM performance systematically degrades in Indic languages due to current benchmark limitations and lack of cultural grounding.

Principles

• Multilingual VLM benchmarks require human verification.
• Culturally grounded content is crucial for VLM evaluation.
• Performance regression is consistent from English to Indic languages.

Method

A framework combining semi-automated translation with human verification generates 4k QA pairs per language across five domains, including adapted English and native Indic datasets.

In practice

• Generate culturally relevant QA pairs for VLM evaluation.
• Utilize HinTel-AlignBench for Indic language VLM assessment.
• Analyze VLM failure modes in multilingual contexts.

Topics

• Vision-Language Models
• Multilingual AI Evaluation
• Indic Languages
• Benchmark Development
• Performance Degradation
• HinTel-AlignBench

Best for: Research Scientist, AI Engineer, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation — VLMs exhibit substantial script-dependent bias, failing to consistently process identical content across different orthographies.
• The 1.6 Billion People Large Language Models Still Can’t Understand — Large language models fail profoundly in many non-English languages, going beyond mere performance degradation to confident incorrectness.
• Diagnosing Visual Ignorance in Vision-Language Models — VLMs' visual ignorance results from internal routing failures and benchmarks that reward language-prior reliance.
• IndicContextEval: A Benchmark for Evaluating Context Utilisation in Audio Large Language Models Across 8 Indic Languages — AudioLLMs vary significantly in how they utilize contextual prompts, with native-script entities and natural language being most effective.
• Polite on the Surface, Wrong in Practice: A Curated Dataset for Fixing Honorific Failures in Multilingual Bangla Generation — Culturally aligned instruction-tuning with BLADE dataset significantly improves honorific consistency in multilingual Bangla LLM generation.
• CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks — CulturALL benchmarks LLMs' multilingual and multicultural reasoning in grounded, real-world tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.