Language Models Learn Constructional Semantics, Not To Mention Syntax: Investigating LM Understanding of Paired-Focus Constructions

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A study investigated the understanding of rare Paired-Focus constructions (e.g., "let alone", "much less") by open-source Language Models. Researchers constructed a novel dataset to evaluate these constructions' meanings, testing both scalar adjectival semantics and general world knowledge. The evaluation involved a wide range of models varying in parameter count, architecture, and pretraining dataset size. Findings indicate that several modestly sized models are sensitive to both the forms and meanings of Paired-Focus constructions. However, models trained on human-scale data consistently failed meaning evaluations. Analysis of training dynamics for open-checkpoint models revealed that Paired-Focus semantic understanding emerges later in training than syntactic knowledge. Furthermore, the acquisition of Paired-Focus semantics correlated with improvements in certain world knowledge domains. These empirical results suggest that modestly sized open-source models can indeed grasp these rare constructions, demonstrating a link between this specific linguistic knowledge and broader semantic understanding.

Key takeaway

For NLP Engineers developing or evaluating language models, this research indicates that even modestly sized open-source models can acquire complex constructional semantics. If you are selecting models for tasks requiring nuanced linguistic understanding, prioritize models that demonstrate robust performance on rare constructions like Paired-Focus. Your evaluation should include tests for semantic understanding, not just syntax, and consider the training stage at which semantic knowledge emerges. This suggests that focusing on models with sufficient training depth and parameter count is crucial for achieving advanced semantic capabilities.

Key insights

Modestly sized open-source LMs can learn rare Paired-Focus construction semantics, with understanding emerging later than syntax and correlating with world knowledge gains.

Principles

• Constructional semantics acquisition is distinct from syntax.
• Semantic understanding correlates with general world knowledge.
• Model size impacts rare construction comprehension.

Method

A novel dataset was constructed to test Paired-Focus construction meanings using scalar adjectival semantics and general world knowledge, evaluating models across parameter counts and pretraining data sizes.

In practice

• Prioritize larger models for nuanced semantic tasks.
• Evaluate LMs on rare linguistic constructions.
• Consider training dynamics for semantic emergence.

Topics

• Language Models
• Constructional Semantics
• Paired-Focus Constructions
• Semantic Understanding
• Training Dynamics
• Open-source Models

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Can Large Language Models Understand Context? — LLMs struggle with nuanced context understanding, especially pre-trained dense models and those with aggressive quantization.
• Floating or Suggesting Ideas? A Large-Scale Contrastive Analysis of Metaphorical and Literal Verb-Object Constructions — Metaphorical and literal language differences are largely construction-specific, not universally consistent.
• [R] Empirical evidence for a primitive layer in small language models — 18 experiments across 4 architectures — Small language models exhibit a consistent activation gap between scaffolding and content semantic primitives.
• Hybrid Human-LLM Corpus Construction and LLM Evaluation for the Caused-Motion Construction — LLMs struggle with complex linguistic constructions like CMC, necessitating novel corpus creation methods.
• Last Call: Build your own Language Model from scratch — The workshop teaches end-to-end language model construction, emphasizing foundational components over API usage.
• How Language Models Learn the Way Humans Do? — Multi-Head Attention allows language models to understand word meaning by simultaneously considering all contextual relationships.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.