Evaluation of Multilingual Ability to Use Spatial Deictic Expressions in Vision-Language Models

2026-07-09 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Natural Language Processing · Depth: Expert, extended

Summary

Researchers developed a multilingual benchmark to evaluate vision-language models' (VLMs) ability to use spatial deictic expressions like "this" and "that" across different languages. Focusing on English, Japanese, Korean, and Chinese, the benchmark adapts the "memory game" paradigm, using 60 Blender-generated images featuring objects at three distinct distances (0.25m, 1.50m, 2.75m). Experiments with Gemma 3 (4B, 12B) and Qwen3-VL (8B, 32B) models revealed that VLMs generally fail to reproduce human-like demonstrative usage, particularly in adjusting selections based on object distance. Models often avoid distal demonstratives in languages with three options (Japanese, Korean), and performance is notably weaker in lower-resource languages like Korean. Qwen3-VL 32B showed relatively better alignment with human distributions in Japanese and Chinese.

Key takeaway

For AI Scientists and Machine Learning Engineers developing multilingual vision-language models, recognize that current models like Gemma 3 and Qwen3-VL significantly diverge from human-like spatial deictic expression usage. Your training strategies should explicitly address distance-dependent demonstrative selection, especially for languages with nuanced spatial distinctions like Japanese and Korean. Prioritize diverse, context-rich pre-training data to improve object recognition and deictic fidelity, as current models struggle to adapt demonstrative choice based on object proximity.

Key insights

Vision-language models struggle to use spatial deictic expressions like humans, especially across diverse languages and distances.

Principles

• VLM demonstrative usage does not align with human distance-based selection.
• Performance in low-resource languages like Korean is significantly weaker.
• Pre-training data impacts VLM object recognition and deictic fidelity.

Method

A VQA task based on the "memory game" paradigm evaluates VLMs' spatial deixis by having them describe objects in Blender-generated images at varying distances.

In practice

• Utilize the provided benchmark for VLM spatial reasoning evaluation.
• Focus VLM training on distance-sensitive deictic expression usage.

Topics

• Vision-Language Models
• Spatial Reasoning
• Spatial Deixis
• Multilingual Evaluation
• Benchmark Development
• Gemma 3
• Qwen3-VL

Code references

• ynklab/multilingual-demonstratives-eval

Best for: AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Evaluation of Multilingual Ability to Use Spatial Deictic Expressions in Vision-Language Models — Vision-language models struggle with human-like multilingual spatial deictic reasoning, particularly distance-based demonstrative selection.
• Evaluation of Multilingual Ability to Use Spatial Deictic Expressions in Vision-Language Models — VLMs struggle with human-like spatial deictic expression usage, particularly distance-based demonstratives, across languages.
• Disparities In Negation Understanding Across Languages In Vision-Language Models — VLMs show affirmation bias, with negation understanding varying significantly across diverse languages and models.
• Systematic Performance Degradation in Indic Vision-Language Models: Evidence from Hindi and Telugu — Multilingual VLM performance systematically degrades in Indic languages due to current benchmark limitations and lack of cultural grounding.
• Why Far Looks Up: Probing Spatial Representation in Vision-Language Models — Vision-language models consistently conflate vertical image position with distance, hindering robust spatial reasoning.
• Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation — VLMs exhibit substantial script-dependent bias, failing to consistently process identical content across different orthographies.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.