To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoning

2026-04-23 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A study investigates the generalization ability of decoder-only transformer models in abstract symbolic reasoning, specifically propositional logic problems. Previous research indicated models struggle with unseen variable names due to difficulty in copying new tokens. This work demonstrates both theoretically and empirically that a representational collapse, where unembeddings of unseen tokens converge to similar vectors during training, also plays a crucial role. This collapse hinders the model's ability to distinguish multiple unseen variables, particularly when embedding and unembedding parameters are shared. The findings provide a mechanistic explanation for the effectiveness of interventions like "active forgetting." The authors propose a combination of architectural changes, data diversity, and freezing or resetting (un)embeddings to achieve generalization to unseen tokens, supported by experiments on propositional logic. Evidence of (un)embedding collapse was also observed in unused tokens within the Gemma 3 family of open-weight models, suggesting poor initialization for finetuning.

Key takeaway

For AI Engineers developing or finetuning transformer models for symbolic reasoning, understanding and mitigating unembedding collapse is critical. Your models may struggle with unseen variables due to this representational issue, even with diverse training data. Implement architectural changes, ensure data diversity, and consider freezing or periodically resetting (un)embeddings to improve generalization to novel tokens and prevent poor finetuning initialization.

Key insights

Unseen token unembedding collapse hinders transformer generalization in symbolic reasoning, explaining prior failures.

Principles

• Representational collapse impacts generalization.
• Shared embeddings exacerbate unseen token issues.

Method

A combination of architectural changes, data diversity, and (un)embedding freezing/resetting improves generalization to unseen tokens in transformers.

In practice

• Consider "active forgetting" for unseen tokens.
• Evaluate (un)embedding initialization for finetuning.

Topics

• Transformers
• Symbolic Reasoning
• Representational Collapse
• Unembedding Collapse
• Generalization Ability

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Student

Related on AIssential

• To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoning — Transformer models exhibit representational collapse of unseen token unembeddings, hindering symbolic reasoning generalization.
• Symbolic Grounding Reveals Representational Bottlenecks in Abstract Visual Reasoning — Symbolic input significantly improves LLM performance on abstract visual reasoning, identifying representation as a key bottleneck.
• Decoder only Transformer : Building a GPT-2 model prototype to make it understand Natural Language… — Training a GPT-2 prototype on classic novels demonstrates decoder-only Transformer architecture and performance benefits of pre-trained weights.
• Pause or Fabricate? Training Language Models for Grounded Reasoning — LLMs can achieve grounded reasoning by learning to pause and clarify when information is incomplete.
• Google is cooking: Beyond the 'Next-Token' Manifold — Transformer models linearize internal representations for flow, but use nonlinear jumps for reasoning.
• Why Transformers Know Where Words Are (And How They Actually Think): RoPE, ALiBi, and the MLP… — While attention mechanisms determine "what to focus on," they are inherently position-blind, necessitating positional encodings.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.