Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation

2024-01-30 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

A new benchmark, PuMVR (Punjabi Multimodal Visual Reasoning), exposes a significant "Script Gap" in state-of-the-art Vision-Language Models (VLMs). It uses 1,000 parallel image-text instances across Punjabi's Gurmukhi, Shahmukhi, and Roman scripts. This challenges the "One Language, One Script" evaluation paradigm. Evaluating 10 VLMs revealed accuracy deltas up to 16.26% (Llama-3.2-11B-Vision). Script Consistency Rates (SCR) were as low as 24.8%. Visual input boosts performance but does not close the orthographic gap. Cross-script in-context transfer proved brittle. McNemar tests confirmed these degradations are statistically robust for 8 of 10 models. The study proposes SCR as a mandatory metric for script-agnostic VLM evaluation to ensure equitable AI access.

Key takeaway

For AI Scientists and ML Engineers deploying multilingual Vision-Language Models, this research highlights a critical blind spot: current evaluations overlook script consistency. You must move beyond the "One Language, One Script" assumption and explicitly test for orthographic robustness using metrics like Script Consistency Rate (SCR). Failing to do so risks deploying models unreliable for billions of multi-script users, leading to fragmented performance and inequitable AI access. Prioritize script-agnostic evaluation for true multilingual competence.

Key insights

VLMs exhibit substantial script-dependent bias, failing to consistently process identical content across different orthographies.

Principles

• The "One Language, One Script" paradigm masks VLM orthographic bias.
• Visual grounding adds performance but does not close script gaps.
• VLM knowledge can be script-locked, hindering cross-script transfer.

Method

PuMVR benchmark uses 1,000 parallel image-reasoning tasks in Gurmukhi, Shahmukhi, and Roman scripts to isolate orthography. It measures Script Accuracy, Script Consistency Rate (SCR), and Performance Delta.

In practice

• Evaluate VLMs using parallel-script benchmarks like PuMVR.
• Incorporate Script Consistency Rate (SCR) as a mandatory metric.
• Assess cross-script transfer efficiency with few-shot learning.

Topics

• Vision-Language Models
• Multilingual AI Evaluation
• Orthographic Bias
• Script Consistency Rate
• PuMVR Benchmark
• Cross-Script Transfer

Code references

• prabhjotschugh/Not-Truly-Multilingual-PuMVR

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation — Current "multilingual" VLMs exhibit a significant "Script Gap," failing identical visual tasks across different scripts of the same language.
• Disparities In Negation Understanding Across Languages In Vision-Language Models — VLMs show affirmation bias, with negation understanding varying significantly across diverse languages and models.
• Cross-Lingual Response Consistency in Large Language Models: An ILR-Informed Evaluation of Claude Across Six Languages — ILR-informed expert judgment reveals interpretable, domain-dependent cross-lingual variation in LLM outputs.
• Why Vision Language Models Ignore What They See [Munawar Hayat] - 758 — Current VLMs struggle with physics-based generation, visual grounding, and multi-person image fidelity, necessitating improved training and architectural approaches.
• Visuals Lie, Consistency Speaks: Disentangling Spatial Attention from Reliability in Vision-Language Models — VLM reliability is primarily predicted by generation dynamics and internal consistency, not visual attention.
• Vision-Language Models vs Human: Perceptual Image Quality Assessment — VLMs can approximate human image quality judgments, but performance varies significantly by perceptual attribute.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.