Similar Predictions, Different Processes: A Multi-Level Comparison of Human and Multimodal LLM Language Prediction

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Social Sciences & Behavioral Studies · Depth: Expert, quick

Summary

A study compared Mandarin speakers and the Qwen2.5-Omni-7B multimodal large language model on language prediction in Mandarin dative constructions using a visual world paradigm (VWP). Experiment 1, an offline cloze-in-VWP task, found both humans and the model were sensitive to structural and prosodic cues, indicating partial output-level alignment. However, the model exhibited a larger structural effect and an atypical prosody pattern. Experiment 2, an online processing analysis using human eye-tracking and model audio-to-image cross-modal attention, revealed divergent time courses. Humans showed structural effects before the contrastive connective, while the model's sensitivity emerged later, after connective onset. These findings demonstrate that output-level and process-level alignment can dissociate, contributing a multi-level human-model comparison methodology and empirical constraints on multimodal LLM cognitive plausibility claims.

Key takeaway

For research scientists evaluating multimodal LLMs for cognitive plausibility, you should not rely solely on output-level alignment. Your assessments must include process-level comparisons, such as attention time courses, to reveal potential dissociations in how models arrive at predictions. This ensures a more robust understanding of model behavior beyond superficial similarities, guiding the development of more human-like AI systems.

Key insights

Human and multimodal LLM language prediction can align in output but diverge in underlying processing mechanisms.

Principles

• Output-level and process-level alignment can dissociate.
• Multimodal LLMs may exhibit atypical prosody processing patterns.
• Structural effects in language prediction can emerge at different times.

Method

A multi-level human-model comparison methodology combines offline cloze tasks with online eye-tracking and model cross-modal attention measures within a visual world paradigm.

In practice

• Use VWP for human-model language processing comparisons.
• Analyze both offline outputs and online processing timelines.
• Evaluate multimodal LLMs for cognitive plausibility.

Topics

• Multimodal LLMs
• Language Prediction
• Cognitive Plausibility
• Visual World Paradigm
• Qwen2.5-Omni-7B
• Human-AI Comparison

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• Ukrainian Visual Word Sense Disambiguation Benchmark — Multimodal LLMs significantly underperform in Ukrainian Visual-WSD compared to English, revealing a critical language resource gap.
• Nobody gets this right — The distinction between language models and world models is diminishing as AI becomes multimodal and adept at abstract mathematical representations.
• Analysis of the Neglect-Zero Effect in Large Language Models — LLMs may not share the human "neglect-zero effect" cognitive bias regarding empty sets.
• Research on LLM alignment as latent discourse-level regimes vs. token-level filtering? — LLM alignment may stem from global latent space geometry and discourse-level regimes, not just token-level filtering.
• Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages — Multimodal LLMs must integrate text, speech, and vision for low-resource languages, moving beyond English-centric benchmarks.
• Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing — LLMs struggle with deep metaphor comprehension, often misinterpreting concepts and relying on lexical associations over context or full syntactic understanding.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.