On the Role of Directionality in Structural Generalization

2026-07-02 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

A new symbolic backend for natural language parsing, redesigned around Combinatory Categorial Grammar (CCG) directed types, significantly improves structural generalization, particularly in tasks involving directional distinctions. This system, using a deterministic CKY parser and a single linear decoder with 30K learnable parameters, achieved 75.9% ± 6.4% LF exact match when paired with a BERT-base encoder. This performance surpasses the previous state-of-the-art AM-Parser, which scored 70.8% ± 4.3%. The gains are highly directional, with the CCG system outperforming AM-Parser by +29.9 percentage points across all 5 SLOG position-shift categories. While AM-Parser excelled in recursive-depth categories, replacing the encoder with DeBERTa-v3-large boosted the CCG system's overall performance to 90.7% ± 4.9%, with the largest gains in recursive-depth tasks, complementing the directionality improvements. This indicates directional representations shift the bottleneck from the symbolic to the neural layer.

Key takeaway

For NLP Engineers developing robust parsers for complex language structures, incorporating explicit directionality into your symbolic representations is crucial. Your current parsing systems, if lacking directional encoding, may struggle with position-shift generalization, as seen with AM-Parser's limitations. Consider adopting CCG-directed type systems to achieve significant gains in structural generalization, and pair them with advanced neural encoders like DeBERTa-v3-large to further enhance performance on recursive-depth tasks.

Key insights

Encoding directionality in symbolic parsing significantly improves structural generalization, especially for position-shift tasks.

Principles

• Directional representations shift parsing bottlenecks to neural encoders.
• CCG directed types enhance structural generalization in parsing.

Method

Redesigning a symbolic backend with CCG directed types, using a deterministic CKY parser and a single linear decoder, to encode directionality for improved parsing.

In practice

• Integrate CCG directed types for tasks requiring directional distinctions.
• Upgrade neural encoders to improve recursive-depth generalization.

Topics

• Structural Generalization
• Natural Language Parsing
• Combinatory Categorial Grammar
• Directional Representations
• BERT-base
• DeBERTa-v3-large

Best for: Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• Structure Before Collapse: Transient semantic geometry in next-token prediction — Language models transiently learn semantic structure from one-hot labels before representations collapse to a symmetric, non-semantic state.
• Grammar as an Injectable: A Trojan Horse to NLP | Towards Data Science — CCG offers a non-statistical, algebraic framework for NLP, treating words as functions to ensure grammatical consistency and enhance interpretability.
• The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledge — W2S generalization enables strong models to learn from weak supervision without catastrophic forgetting of other capabilities.
• Structural Generalization on SLOG without Hand-Written Rules — A neural cellular automaton can achieve structural generalization in semantic parsing without hand-written rules.
• Relational Linear Properties in Language Models: An Empirical Investigation — A new Kullback-Leibler divergence probing method efficiently tests relational linearity in language models, showing variations across models and sensitivity to phrasing.
• Code-Switching Reveals Language Anchoring in Multilingual LLMs — Language anchoring in MLLMs, revealed by code-switching, causes performance degradation, which can be mitigated by steering hidden states.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.