Examining Large Language Models’ form-meaning mappings of information structure constructions in Mandarin Chinese

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

A study by Shihui Li, Xiaojuan Tan, and Jelke Bloem investigates Large Language Models' linguistic competence with Mandarin Chinese information structure constructions, specifically the "ba" (把, disposal) and "bei" (被, passive) constructions. While Construction Grammar (CxG) knowledge in LLMs is well-studied for English, its application to other languages remains underexplored. These Mandarin constructions are crucial for managing information structure, foregrounding topical elements, and encoding systematic form-meaning mappings related to the object's semantic role. Researchers developed a new minimal-pair dataset with seven paradigms to probe models on both syntactic constraints and verb–construction compatibility. Results indicate that many models face challenges in capturing the underlying form-meaning mappings of the "ba" construction, even as they achieve high accuracy on paradigms driven by surface syntactic cues.

Key takeaway

For NLP Engineers developing or evaluating Large Language Models for Mandarin Chinese, you should prioritize testing beyond surface syntactic accuracy. This research indicates that models often struggle with the "ba" construction's complex form-meaning mappings, even when surface cues are handled correctly. Ensure your evaluation metrics and training data specifically address deep semantic understanding and Construction Grammar principles to build truly competent multilingual LLMs.

Key insights

LLMs demonstrate limited understanding of Mandarin "ba" construction's form-meaning mappings, despite handling surface syntax.

Principles

• LLM linguistic competence varies by language.
• Surface syntactic cues are insufficient for deep CxG.
• Form-meaning mappings challenge LLMs.

Method

A new minimal-pair dataset was constructed, comprising seven paradigms. This dataset targets both syntactic constraints and verb–construction compatibility to probe LLM linguistic competence.

In practice

• Evaluate LLMs on form-meaning mappings.
• Develop datasets for specific language CxG.
• Prioritize semantic role understanding in LLMs.

Topics

• Large Language Models
• Mandarin Chinese
• Construction Grammar
• Information Structure
• Linguistic Competence
• Minimal-Pair Datasets

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Hybrid Human-LLM Corpus Construction and LLM Evaluation for the Caused-Motion Construction — LLMs struggle with complex linguistic constructions like CMC, necessitating novel corpus creation methods.
• ChLogic: Evaluating Robustness of Logical Reasoning in Chinese Expressions — ChLogic reveals a persistent English-Chinese logical reasoning gap in LLMs, influenced by Chinese surface forms and translation artifacts.
• When Similar Means Different: Evaluating LLMs on Arabic--Hebrew Cognates — Current LLMs struggle with cross-lingual semantic disambiguation, especially false friends and loanwords, due to over-reliance on surface forms.
• Phun-Bench: Evaluating LLMs on Phonological Understanding in Chinese — LLMs struggle with flexible phonological understanding despite recalling pronunciations, indicating a research gap.
• LLMs for automatic annotation of Mandarin narrative transcripts — LLMs can automate complex discourse-level annotation in non-English speech, significantly reducing time.
• Datasets for Verb Alternations across Languages: BLM Templates and Data Augmentation Strategies — This work addresses the underexplored ability of large language models (LLMs) to capture cross-sentence paradigmatic patterns, specifically verb alternations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.