Evaluation of Multilingual Ability to Use Spatial Deictic Expressions in Vision-Language Models

2026-07-08 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new benchmark evaluates the multilingual ability of Vision-Language Models (VLMs) to use spatial deictic expressions, such as "this" and "that." These expressions require VLMs to perform joint reasoning across language and visual space, grounding context-dependent references within an image's spatial structure. Furthermore, VLMs must understand language-specific spatial distinctions to select appropriate deictic expressions across different languages. Developed for four languages, the benchmark was used in experiments published on 2026-07-08. The findings indicate that the evaluated models utilize demonstratives in a manner distinct from human usage, particularly concerning the selection of appropriate demonstratives based on the object's distance.

Key takeaway

For Machine Learning Engineers developing multilingual Vision-Language Models, recognize that current models exhibit significant differences from human behavior in using spatial deictic expressions. Your VLM implementations may struggle with context-dependent references and distance-based demonstrative selection across languages. Prioritize research and development efforts on improving VLMs' ability to ground spatial references and understand language-specific spatial distinctions for more natural and accurate human-VLM interaction.

Key insights

VLMs struggle with human-like spatial deictic expression usage, particularly distance-based demonstratives, across languages.

Principles

• VLMs need joint language and visual spatial reasoning.
• Language-specific spatial distinctions are crucial for deictic expressions.

Method

A benchmark was developed to evaluate multilingual VLM ability to use spatial deictic expressions in four languages, focusing on context-dependent references.

Topics

• Vision-Language Models
• Spatial Reasoning
• Deictic Expressions
• Multilingual AI
• VLM Benchmarking
• Natural Language Understanding

Best for: AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Evaluation of Multilingual Ability to Use Spatial Deictic Expressions in Vision-Language Models — Vision-language models struggle with human-like multilingual spatial deictic reasoning, particularly distance-based demonstrative selection.
• Disparities In Negation Understanding Across Languages In Vision-Language Models — VLMs show affirmation bias, with negation understanding varying significantly across diverse languages and models.
• The Dual Mechanisms of Spatial Variable Binding in Vision-Language Models — VLMs use dual spatial reasoning mechanisms, with the vision encoder providing the dominant, globally distributed ordering information.
• Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation — VLMs exhibit substantial script-dependent bias, failing to consistently process identical content across different orthographies.
• Why Far Looks Up: Probing Spatial Representation in Vision-Language Models — Vision-language models consistently conflate vertical image position with distance, hindering robust spatial reasoning.
• Systematic Performance Degradation in Indic Vision-Language Models: Evidence from Hindi and Telugu — Multilingual VLM performance systematically degrades in Indic languages due to current benchmark limitations and lack of cultural grounding.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.