Exposure is Optional: Learning Unlike Coordination in Language Models

2026-07-23 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, extended

Summary

Research by Jiamu Luo and Shane Steinert-Threlkeld investigates whether language models acquire "unlike coordination"—combining syntactically different constituents—through direct exposure or general compositional abilities. Using Filtered-Corpus Training (FiCT), they trained GPT-2 models on corpora where all instances of unlike coordination were systematically removed. The study found that direct exposure is not necessary; models trained on filtered data successfully generalized to unlike coordination, achieving perplexity and grammaticality judgments comparable to models trained on unfiltered text. Internal representation analyses revealed that GPT-2 processes unlike coordination via two mechanisms: treating conjoined elements as similar structural categories (supercategory) or through a deletion-like process. Both mechanisms appear learnable from exposure to alike coordination alone, challenging the necessity of the Law of Coordination of Likes (LCL) in linguistic theory.

Key takeaway

For NLP Engineers designing training regimens or AI Scientists evaluating model linguistic competence, this research indicates that language models can acquire complex syntactic structures like "unlike coordination" without direct training examples. You should consider that models generalize from indirect evidence, suggesting that meticulously filtering out every instance of a target phenomenon might not prevent its acquisition. Instead, focus on providing rich, diverse "alike coordination" examples and employ linguistically grounded evaluation tests to accurately probe the nuances of learned syntactic constraints and internal processing mechanisms.

Key insights

Language models generalize complex "unlike coordination" from "alike coordination" without direct exposure, challenging strict syntactic rules.

Principles

• LMs do not inherently presuppose a strict Law of Coordination of Likes (LCL).
• General compositional abilities enable generalization of complex syntactic structures.
• Models process unlike coordination via "supercategory" or "deletion" mechanisms.

Method

Filtered-Corpus Training (FiCT) involves training GPT-2 models from scratch on corpora systematically stripped of specific linguistic structures (e.g., unlike coordination) to isolate learning conditions.

In practice

• Design training data to exploit indirect evidence for complex linguistic structures.
• Use linguistically grounded evaluations to interpret nuanced model behavior.

Topics

• Language Models
• Syntactic Generalization
• Unlike Coordination
• Filtered-Corpus Training
• GPT-2
• Linguistic Theory

Best for: Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• Exposure is Optional: Learning Unlike Coordination in Language Models — Language models can acquire complex linguistic structures, such as unlike coordination, without direct training data exposure.
• Syntactic Priming in Few-Shot Learning: How Demonstration Structure Shapes LLM Performance — LLMs exhibit strong syntactic priming from few-shot demonstrations, influencing output structure beyond surface-level matching.
• Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models — Task competence and instruction following are distinct abilities, with standard accuracy hiding instruction-following failures.
• Harnessing Linguistic Dissimilarity for Language Generalization on Unseen Low-Resource Varieties — Linguistic dissimilarity is a crucial cue for generalizing to unseen low-resource language varieties.
• Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining — Language models can naturally "ungrok" learned rules during pretraining, dictated by the rule's support frequency in the corpus.
• Language Acquisition Device in Large Language Models — LAD-inspired pre-pretraining with MP-Struct improves LLM data efficiency and instills human-like linguistic biases.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.