What do Language Models Learn and When? The Implicit Curriculum Hypothesis

2026-04-09 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Research introduces the Implicit Curriculum Hypothesis, proposing that large language model pretraining follows a compositional and predictable skill acquisition order across different models and data. To test this, a suite of simple, composable tasks covering retrieval, morphological transformations, coreference, logical reasoning, and mathematics was designed. Tracking skill emergence points across four model families ranging from 410M to 13B parameters revealed a striking consistency in emergence orderings, with a Pearson correlation coefficient of $ρ= .81$ across 45 model pairs. Composite tasks consistently emerged after their component tasks. The study also found that this structural emergence is encoded in model representations, where tasks with similar function vector representations exhibit similar training trajectories. This allows for effective prediction of training trajectories for simple held-out compositional tasks, achieving an $R^2$ of $.68$-".84" across models.

Key takeaway

For research scientists developing or evaluating large language models, understanding the Implicit Curriculum Hypothesis is crucial. Your model's pretraining likely follows a predictable, compositional skill acquisition path, meaning you can anticipate which capabilities will emerge and in what order. This insight allows you to design more efficient training regimes and interpret model behavior by analyzing internal representations to predict learning trajectories, potentially reducing extensive evaluation cycles.

Key insights

LLM pretraining follows a consistent, compositional skill emergence order, predictable from internal representations.

Principles

• Skill emergence is compositional.
• Emergence order is consistent across models.
• Model representations encode skill trajectories.

Method

A task suite tracks skill emergence across model families, correlating emergence order with representation similarity to predict training trajectories.

In practice

• Design curricula for specific skills.
• Analyze model representations for skill encoding.
• Predict task learning without full evaluation.

Topics

• Large Language Models
• LLM Pretraining
• Implicit Curriculum Hypothesis
• Skill Emergence
• Model Representations

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• What Kind of Language is Easy to Language-Model Under Curriculum Learning? — Curriculum learning significantly impacts language models' inductive bias, influencing their ability to learn typological language patterns.
• Rethinking Pre-Training for Agentic AI with Aakanksha Chowdhery - #759 — Agentic AI requires fundamental pre-training shifts, not just post-training, to enable multi-step reasoning and dynamic tool use.
• [R] Empirical evidence for a primitive layer in small language models — 18 experiments across 4 architectures — Small language models exhibit a consistent activation gap between scaffolding and content semantic primitives.
• Therefore I am. I Think — LLMs can encode action choices before textual deliberation, shaping subsequent chain-of-thought reasoning.
• Language Acquisition Device in Large Language Models — LAD-inspired pre-pretraining with MP-Struct improves LLM data efficiency and instills human-like linguistic biases.
• When Do Attention Circuits Form? Developmental Trajectories of Capability and Attention-Sink Emergence Across Three 1B-ClassArchitectures — Induction and attention-sink circuit formations are distinct developmental transitions in 1B-class language models.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.