LLMs and their Limited Theory of Mind: Evaluating Mental State Annotations in Situated Dialogue

2026-07-20 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A novel, two-step framework utilizes large language models (LLMs) to track Shared Mental Models (SMM) within team dialogues. This framework positions LLMs both as annotators for monitoring a team's SMM and as automated detectors for discrepancies in individual mental states. An SMM coherence evaluation framework was applied to six dialogues from a previously published team corpus, generating a dataset of human and LLM SMM annotations. The empirical assessment of LLM-based discrepancy detection revealed that while LLMs demonstrate apparent coherence on simple natural-language annotation tasks, they systematically make errors when spatial reasoning or disambiguation of transcription-level disfluencies are required.

Key takeaway

For NLP Engineers evaluating LLMs for complex dialogue analysis or Shared Mental Model tracking, you should recognize their current limitations. While LLMs perform well on basic natural-language annotation, your systems will likely fail in tasks requiring spatial reasoning or disambiguation of transcription-level disfluencies. Consider hybrid approaches or human-in-the-loop systems for these challenging scenarios to ensure robust SMM coherence.

Key insights

LLMs can track Shared Mental Models in dialogue but struggle with complex reasoning like spatial understanding or disfluency disambiguation.

Principles

• LLMs show coherence on straightforward natural-language annotation.
• LLMs err in spatial reasoning or disfluency disambiguation.

Method

The framework uses LLMs as annotators to track team SMM and as detectors for discrepancies among individual mental states as represented in individual mental models.

In practice

• Apply LLMs for basic dialogue annotation tasks.
• Avoid LLMs for SMM tracking involving spatial context.

Topics

• Large Language Models
• Shared Mental Models
• Dialogue Annotation
• Discrepancy Detection
• Spatial Reasoning
• Natural Language Processing

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Reasoning as Pattern Matching: Shared Mechanisms in Human and LLM Everyday Reasoning — Human and LLM everyday reasoning aligns with content-sensitive pattern-matching, challenging abstract world model assumptions.
• Conversational Grounding in Large Language Models: Evaluation Methods, Challenges and Future Directions — Evaluating conversational grounding in LLMs requires both explicit and implicit methods, facing significant methodological challenges.
• Exploring Coherence of LLMs in Multilingual Question Answering — LLM coherence in multilingual QA is not solely tied to model size or accuracy, varying significantly across languages.
• Do Large Language Models Possess a Theory of Mind? A Comparative Evaluation Using the Strange Stories Paradigm — GPT-4o demonstrates human-comparable Theory of Mind capabilities in text-based evaluations, outperforming smaller LLMs.
• Unveiling the Limits of Large Language Models in Inferring Pragmatic Meaning from Non-Verbal Responses — LLMs significantly struggle to infer pragmatic meaning from non-verbal communication, unlike their performance with verbal cues.
• Beyond Sally-Anne: Evaluating Theory of Mind in LLMs using Epistemic Schelling Points — The Epistemic Asymmetry Schelling Task (EAST) reveals LLMs' significant functional social reasoning and epistemic tracking gaps.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.