Smooth Scaling Laws Hide Stepwise Token Learning

2026-07-13 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

This paper introduces a token-level framework to explain why language model loss exhibits power-law scaling laws, moving beyond aggregate loss explanations. Researchers decomposed scaling laws into localized learning events of individual contextualized tokens, fitting their loss trajectories with sigmoid functions. This revealed that token learning is concentrated in specific transitions, creating a "learning-time spectrum" that dictates the overall scaling-law shape. Across over 100 pre-training runs on diverse real-language corpora, involving models up to 6B parameters and 300B training tokens, this spectrum quantitatively reconstructed the validation loss derivative along training-step, data-scale, and model-scale axes. Furthermore, by reshaping the training distribution based on when tokens become learnable, the method achieved an 11% faster validation-loss reduction, demonstrating the signal's practical utility.

Key takeaway

For Machine Learning Engineers optimizing large language model pre-training, understanding token-level learning dynamics is crucial. Your team can utilize the "learning-time spectrum" to predict and control scaling behavior. By estimating when tokens are learned and strategically reshaping your training data distribution to prioritize specific learning intervals, you can achieve measurable improvements, such as an 11% acceleration in validation-loss reduction, leading to more efficient compute allocation and faster model development.

Key insights

Language model scaling laws are primarily governed by the distribution of token-level learning times, not individual token loss shapes.

Principles

• Token learning occurs as localized, sigmoid-like transitions.
• Learning pulse shapes are largely consistent across tokens.
• Macroscopic loss decay is a superposition of token learning events.

Method

Decompose aggregate loss derivative L'(a) into a convolution (p*g)(a), where g is the average centered learning pulse and p(τ) is the empirical learning-time spectrum derived from sigmoid fits of token loss trajectories.

In practice

• Estimate token learning times via sigmoid fitting.
• Reshape training data by upweighting samples with target learning times.
• Accelerate validation-loss reduction by optimizing data distribution.

Topics

• Language Model Scaling Laws
• Token-Level Learning Dynamics
• Learning-Time Spectrum
• Data Distribution Optimization
• LLM Pre-training
• Sigmoid Fitting

Code references

• applewpj/token-learning-spectrum

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• A Quantitative Experimental Repeated Measures Study of Training Dynamics in a Small Llama Style Language Model Under a Compute-Aware Token Budget — Compute-constrained LLM training trajectories reveal non-monotonic degradation and instability, challenging endpoint-only evaluation.
• Scaling Laws, Carefully — Scaling laws predict deep learning loss based on model size, data, and compute, but optimal allocation depends critically on experimental scale and data availability.
• Structure Before Collapse: Transient semantic geometry in next-token prediction — Language models transiently learn semantic structure from one-hot labels before representations collapse to a symmetric, non-semantic state.
• How Does a Large Language Model Actually Learn? — LLMs acquire complex capabilities through iterative next-token prediction and error-driven weight adjustments.
• Scaling Laws for Classical Machine Learning on Tabular Data: A Benchmark Study — Classical ML models on tabular data exhibit predictable power-law scaling, but implementation variance and irreducible error are critical.
• Explaining Data Mixing Scaling Laws — A unified framework explains data mixing scaling laws through capacity competition and noise reduction.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.