Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases

2026-05-22 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

A study investigates the "small-vs-large gap," a counterintuitive phenomenon where training on smaller datasets with increased repetitions leads to faster learning and reduced computational cost compared to larger datasets. This effect is observed across various algorithmic tasks, architectures like MLPs and Transformers, and optimizers, and cannot be fully explained by existing theories such as CSQ-SQ differences or gradient variance reduction. The research posits that this speedup stems from favorable optimization biases induced by dataset sampling, which promotes appropriate layer-wise growth. Empirical evidence, including a 100x compute speedup for (20,6)-sparse parity using N=2^14 samples versus N=2^20, and theoretical analysis (Theorem 1) confirm that smaller datasets reduce convergence steps. The speedup even persists with random labels and full-batch gradient updates, highlighting sampling bias as a key mechanism.

Key takeaway

For machine learning engineers optimizing training compute, you should proactively consider using smaller datasets with increased repetitions, especially for reasoning tasks. This strategy, which utilizes beneficial sampling biases for faster layer-wise growth, can significantly reduce total compute, as demonstrated by 100x speedups. You might also explore multi-phase training schedules to balance rapid initial learning with robust generalization.

Key insights

Smaller datasets with repetition accelerate learning by inducing beneficial sampling biases that adjust layer-wise growth.

Principles

• Sampling biases in smaller datasets act as favorable inductive biases.
• Layer-wise growth adjustment is key to faster feature learning.
• The small-vs-large gap is not solely due to stochastic gradient properties.

Method

The paper analyzes a 2-phase training process for 2-layer networks with quadratic activation, using correlation loss and projected updates. Phase 1 uses a small dataset to grow the outer weight, followed by Phase 2 with full population gradients.

In practice

• Proactively use smaller datasets with more repetitions for reasoning tasks.
• Adjust layer-wise initialization or learning rates to mitigate the gap.
• Consider multi-phase training for balancing speed and generalization.

Topics

• Small-vs-large Gap
• Dataset Sampling Bias
• Machine Learning Optimization
• Deep Learning Training
• Transformers
• Multi-phase Training

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• Repetition Mismatch: Why Data Mixture Experiments Don't Scale and How to Fix Them — Repetition mismatch, where data repetition rates change with scale, causes small-scale data mixture experiments to fail, but repetition control fixes it.
• Learning from a Biased Sample — Sampling bias can be mitigated by learning decision rules robust to worst-case risk under a conditional Γ-biased sampling model.
• Learning, Fast and Slow: Towards LLMs That Adapt Continually [R] — Combining slow parameter updates with fast context optimization enhances LLM adaptation and mitigates forgetting.
• Training neural networks faster without GPU [RB] (Ep. 77) — Data echoing speeds neural network training by replicating data to optimize CPU utilization, especially when upstream tasks bottleneck the pipeline.
• Is Backpropagation Optimal? When Synthetic Gradients Improve Sample Efficiency — Synthetic gradients can offer superior sample efficiency over backpropagation, challenging its default status in neural network training.
• Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention — Larger models learn more by reducing neural interference, allowing retention of rare-task features.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.