Exposure is Optional: Learning Unlike Coordination in Language Models

2026-07-22 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A study investigates whether language models require direct exposure to "unlike coordination" during training or if this linguistic structure emerges from general compositional abilities. Using Filtered-Corpus Training (FiCT), researchers trained GPT-2 models on datasets from which all instances of unlike coordination had been removed. The findings indicate that direct exposure is not necessary; models trained on filtered data successfully generalized to unlike coordination, achieving perplexity and grammaticality judgments comparable to those trained on unfiltered text. Furthermore, analyses of internal representations suggest that language models process unlike coordination by treating conjoined elements as belonging to similar structural categories or through a deletion-like mechanism. Both processing methods appear learnable from exposure to alike coordination alone, contributing to understanding LM linguistic structure representation.

Key takeaway

For NLP Engineers developing or evaluating language models, this research suggests that your training data may not require exhaustive coverage of every complex linguistic phenomenon like unlike coordination. You can potentially achieve robust generalization by focusing on foundational compositional abilities and exposure to related linguistic structures. This approach could streamline data curation efforts and improve model efficiency in handling novel syntactic patterns.

Key insights

Language models can acquire complex linguistic structures, such as unlike coordination, without direct training data exposure.

Principles

• LMs generalize complex syntax from related structures.
• Direct exposure to every linguistic form is not required.
• Internal processing adapts for novel coordination types.

Method

Filtered-Corpus Training (FiCT) involves removing specific linguistic instances, like unlike coordination, from training data to test models' generalization capabilities.

In practice

• Train models on datasets without exhaustive linguistic examples.
• Focus training on core compositional linguistic abilities.
• Evaluate LM generalization to novel syntactic patterns.

Topics

• Language Models
• Linguistic Coordination
• GPT-2
• Syntactic Generalization
• Filtered-Corpus Training
• Natural Language Processing

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Exposure is Optional: Learning Unlike Coordination in Language Models — Language models generalize complex "unlike coordination" from "alike coordination" without direct exposure, challenging strict syntactic rules.
• Can AI Save India’s Dying Languages? | Building a Tulu LLM Without Training Data Ft. Prathamesh — Structured prompting and custom romanization enable LLMs to generate low-resource languages without extensive training data.
• On the Robustness of Morphosyntactic Transformation with Large Language Models: The Case of Quechua Collao — LLM morphosyntactic transformation robustness in Quechua Collao depends on model, context, and linguistic complexity.
• Beyond a Global Norm: Personalizing Toxicity Sensitivity in Language Models Without Retraining — Training-free methods can personalize LM toxicity sensitivity, but a trade-off exists with general language quality.
• Language Acquisition Device in Large Language Models — LAD-inspired pre-pretraining with MP-Struct improves LLM data efficiency and instills human-like linguistic biases.
• Do Large Language Models Acquire Phrase-Based Processing? Evidence from Eye Movements and Model-Brain Alignment After Fine-Tuning — Targeted fine-tuning enables LLMs to acquire human-like phrase-based processing, improving model-brain alignment.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.