Evaluating Adjective-Noun Compositionality in LLMs: Functional vs Representational Perspectives

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, short

Summary

Ruchira Dhar, Qiwei Peng, and Anders Søgaard's 2026 paper, presented at *SEM 2026, investigates adjective-noun compositionality in large language models (LLMs). The research employs two distinct evaluation setups: a prompt-based functional assessment and a representational analysis of the models' internal states. Findings reveal a significant discrepancy: while LLMs reliably develop compositional representations internally, this inherent capability does not consistently translate into successful functional task performance across various model variants. This striking divergence highlights the critical importance of contrastive evaluation methods to achieve a more complete and nuanced understanding of LLM capabilities, moving beyond superficial task success metrics.

Key takeaway

For NLP Engineers evaluating LLMs for complex language understanding, you should not rely solely on functional task performance. Your evaluation strategy must incorporate representational analysis of internal model states to uncover true compositional abilities. This contrastive approach will provide a more complete understanding of an LLM's strengths and weaknesses, guiding your model selection and fine-tuning efforts more effectively.

Key insights

LLMs exhibit a gap between internal compositional representations and external task performance.

Principles

• Compositionality is fundamental to language abilities.
• Internal model states can diverge from task success.
• Contrastive evaluation offers deeper model understanding.

Method

Evaluate adjective-noun compositionality using prompt-based functional assessment and representational analysis of internal model states.

In practice

• Assess LLMs with functional tasks.
• Analyze internal model representations.
• Combine evaluation perspectives.

Topics

• Large Language Models
• Compositionality
• Adjective-Noun Phrases
• Model Evaluation
• Representational Analysis
• Natural Language Processing

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing — LLMs struggle with deep metaphor comprehension, often misinterpreting concepts and relying on lexical associations over context or full syntactic understanding.
• Evaluating Generative AI: Did Astral Codex Ten Win His Bet on AI Progress? — Human evaluation reveals current image generation models still struggle with complex compositional prompts, despite scaling improvements.
• MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights — Multilingual LLM evaluation requires fine-grained, agentic diagnosis to transform scores into actionable insights, addressing cultural and deployment realities.
• Principles of Concept Representation in Sentence Encoders — Representational compositionality predicts sentence encoder concept representation capabilities and structural limitations.
• Rethinking Molecular Text Representations for LLMs: An Empirical Study — LLM molecular competence is strongly representation-dependent, with no single representation optimal across all tasks.
• Rhetorical Questions in LLM Representations: A Linear Probing Study — LLMs represent rhetorical questions heterogeneously, with multiple linear directions capturing distinct rhetorical cues.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.