How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size

2026-07-01 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new "three-term scaling law" is introduced, designed to optimize token allocation during large language model training. This law explicitly incorporates model size, total training data, training steps, and batch size. Researchers found that this proposed law accurately predicts the scaling behavior of optimal batch sizes. A significant advantage is its ability to be robustly fitted using a considerably smaller number of training runs, as it effectively utilizes data from runs with suboptimal batch sizes. Furthermore, the three-term law can be employed to derive scaling laws for suboptimal batch sizes and aligns with prior empirical observations concerning the critical batch size in model training.

Key takeaway

For Machine Learning Engineers optimizing large language model training, this three-term scaling law offers a more efficient approach to resource allocation. You can utilize its ability to predict optimal batch sizes and derive scaling laws even from suboptimal training runs. This means you can achieve robust scaling law fitting with significantly fewer experimental runs, accelerating your model development and reducing computational costs.

Key insights

The three-term scaling law optimizes token allocation by integrating model size, training steps, and batch size, enabling robust fitting with fewer runs.

Principles

• Scaling laws can incorporate training steps and batch size.
• Suboptimal batch size data improves scaling law robustness.
• Optimal batch size scaling can be recovered.

Method

The paper proposes fitting a three-term scaling law that explicitly splits training data into training steps and batch size, using a large set of training runs, including those with suboptimal batch sizes.

In practice

• Use the three-term law for optimal batch size prediction.
• Derive scaling laws for suboptimal batch sizes.
• Reduce training runs needed for scaling law fitting.

Topics

• Scaling Laws
• Large Language Models
• Training Optimization
• Batch Size Optimization
• Token Allocation
• Model Training Efficiency

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Optimal Splitting of Language Models from Mixtures to Specialized Domains — Optimizing compute allocation between general pretraining and specialized continued pretraining improves language model performance.
• 🚀 Scaling to LLMs: Why Bigger Models Get Smarter — LLM performance scales predictably with compute, parameters, and data, unlocking emergent abilities at certain thresholds.
• Compute Optimal Tokenization — Optimal language model scaling depends on data in bytes, not tokens, with a compute-dependent optimal compression rate.
• Scaling Capability in Token Space: An Analysis of Large Vision Language Model — LVLMs exhibit predictable sublinear and linear scaling regimes based on vision token count.
• Token-Operations-Oriented Inference Optimization Techniques for Large Models — Token-oriented inference optimization requires a four-layer architecture to balance quality, cost, and stability for large model services.
• DeepSeek-V3: A Large-Scale MoE Pretraining Benchmark for MLPerf Training v6.0 — MLPerf Training v6.0 now benchmarks MoE LLMs, reflecting industry shifts to sparse computation and specialized architectures.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.