Hybrid Human-LLM Corpus Construction and LLM Evaluation for the Caused-Motion Construction

2026-05-04 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

A novel pipeline has been developed for constructing linguistically annotated corpora and evaluating Large Language Models (LLMs) on complex grammatical structures, specifically the caused-motion construction (CMC), exemplified by "She sneezed the foam off her cappuccino." This research addresses the absence of adequate Construction Grammar (CxG) corpora by leveraging dependency parsing and LLMs to significantly reduce annotation costs for rare linguistic phenomena. The pipeline was used to evaluate OpenAI, Gemma3, Llama3, OLMo2, Mistral, and Aya models. The findings indicate that most tested LLMs struggle to fully comprehend the motion component that the CMC adds to a sentence, suggesting a persistent challenge in their understanding of nuanced linguistic meaning.

Key takeaway

For NLP Engineers and Research Scientists evaluating LLM linguistic capabilities, this work highlights that current models, including OpenAI, Gemma3, and Llama3, struggle with nuanced constructions like caused-motion. You should consider incorporating specialized linguistic tests beyond standard benchmarks to thoroughly assess an LLM's understanding of complex semantics. Furthermore, explore integrating NLP-assisted corpus construction methods to efficiently build targeted datasets for such evaluations.

Key insights

LLMs struggle with complex linguistic constructions like CMC, necessitating novel corpus creation methods.

Principles

• Constructions carry meaning beyond individual words.
• NLP and LLMs can reduce annotation costs for rare linguistic phenomena.

Method

A novel NLP-assisted pipeline uses dependency parsing and LLMs to collect and linguistically annotate text, enabling evaluation of LLMs on specific constructions like CMC.

In practice

• Apply dependency parsing for cost-effective linguistic annotation.
• Evaluate LLMs on specific, complex grammatical constructions.

Topics

• Caused-Motion Construction
• Construction Grammar
• Large Language Models
• Corpus Construction
• Linguistic Annotation
• LLM Evaluation
• Dependency Parsing

Best for: AI Engineer, Machine Learning Engineer, AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• Linguistic Issues in Computational Linguistics for Non-English Languages — English-centric LLM design creates systematic linguistic weaknesses for non-English languages across multiple levels.
• A Scalable Tool for Measuring Manner and Result Verbs in Developmental Language Research — A RoBERTa classifier, trained on LLM-generated annotations, accurately distinguishes manner and result verbs at scale.
• LLMs for automatic annotation of Mandarin narrative transcripts — LLMs can automate complex discourse-level annotation in non-English speech, significantly reducing time.
• Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling — NL-to-FOL benchmarks contain significant errors, distorting model evaluation, but LLM-assisted review can efficiently correct them.
• A Larger Annotated Corpus of Portuguese Coreference — LLMs can annotate Portuguese coreference, but a new LLM-based system did not outperform existing rule-based methods.
• Beyond Surface Forms: A Comprehensive, Mechanism-Oriented Taxonomy of Indirect Linguistic Encoding for LLM-Based Coded Language Detection — A mechanism-oriented taxonomy of indirect linguistic encoding significantly enhances LLM-based coded language detection.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.