Nothing from Something: Can a Language Model Discover 0?

2026-06-15 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

AI systems, particularly language models, are being explored for their capacity to extend human mathematical knowledge beyond their training data, specifically regarding out-of-distribution generalization. A study examined whether modern AI models could independently discover the concept of "zero" using simple arithmetic as a case study. The research found that language models of a GPT-2 size are unable to perform this generalization at test time, irrespective of prior language pretraining. However, these models significantly improve after training on tens or hundreds of examples of zero. Furthermore, language pretraining was observed to reduce the number of required examples by approximately 50%, indicating that language abilities can scaffold mathematical discovery in neural models. This suggests a pathway for AI to expand its mathematical horizons.

Key takeaway

For AI scientists developing models for advanced mathematical reasoning, you should recognize that current language models struggle with genuinely novel concept discovery like "zero" without explicit exposure. Prioritize few-shot training on new mathematical structures, as even tens or hundreds of examples significantly improve generalization. Additionally, utilize language pretraining, as it can reduce the required training data by approximately 50%, accelerating your model's ability to expand its mathematical horizons.

Key insights

Language models struggle to discover zero independently but improve with few-shot training, especially with pretraining.

Principles

• Mathematical discovery requires strong OOD generalization.
• Language abilities can scaffold mathematical learning.
• Few-shot training enables concept acquisition.

Method

Evaluated GPT-2 size language models' ability to discover zero using simple arithmetic, testing generalization before and after few-shot training on zero examples.

In practice

• Train models on minimal examples for new concepts.
• Utilize language pretraining for mathematical tasks.
• Test OOD generalization for novel mathematical structures.

Topics

• Language Models
• Mathematical Discovery
• OOD Generalization
• Concept Learning
• GPT-2 Architecture
• Few-shot Learning

Best for: Research Scientist, AI Scientist, AI Student

Related on AIssential

• Small Initialization Matters for Large Language Models — Small parameter initialization significantly improves large language model pretraining and reasoning capacity by guiding distinct developmental trajectories.
• OmniGCD: Abstracting Generalized Category Discovery for Modality Agnosticism — OmniGCD enables zero-shot, modality-agnostic category discovery by decoupling representation learning from category discovery.
• What Really Improves Mathematical Reasoning: Structured Reasoning Signals Beyond Pure Code — Code alone doesn't generalize reasoning; structured cross-domain traces and dense math samples are key for mathematical reasoning in LMs.
• Grounding vs. Compositionality: On the Non-Complementarity of Reasoning in Neuro-Symbolic Systems — Symbol grounding alone is insufficient for compositional generalization; explicit reasoning objectives are required.
• The Evolution of Reasoning in Small Language Models [Yejin Choi] - 761 — Focused investment and innovative data strategies can significantly enhance small language models' reasoning capabilities and address AI homogeneity.
• The easiest way to improve prompts — Zero-shot and few-shot prompting are core techniques for controlling language model output based on guidance provided.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.