When Does Generating More Help? Disentangling Fixed-Source Synthesis from Source Expansion in Synthetic Data Scaling

2026-07-02 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

The paper "When Does Generating More Help? Disentangling Fixed-Source Synthesis from Source Expansion in Synthetic Data Scaling" investigates two distinct approaches to scaling synthetic data: Source Expansion (SE), which involves adding seed materials or generators, and Fixed-Source Synthesis (FSS), which scales the generation budget while keeping the source fixed. The authors highlight that prior research often conflates these methods, leaving FSS underexplored. They isolate FSS by maintaining a fixed seed-question pool and teacher model, varying only the per-question response budget using Rejection Sampling. An adapted rectified scaling law, derived from repeated sampling, accurately predicts performance at high budgets based on low-budget fits. Empirically, SE and FSS perform comparably at small budgets, but SE, by adding seed questions, surpasses FSS at large, matched total-sample budgets. Within FSS, neither synthesizing more questions nor altering the protocol outperforms plain Rejection Sampling. This establishes FSS as a bounded scaling axis and a controlled environment for comparing synthesis protocols.

Key takeaway

For AI Scientists scaling synthetic data generation, recognize that Fixed-Source Synthesis (FSS) offers bounded performance gains. If your goal is large-scale data, prioritize Source Expansion (SE) by adding more seed materials, as it outperforms simply generating more responses from a fixed source at higher budgets. Conversely, FSS provides a controlled environment for rigorously comparing different synthesis protocols, especially when working with smaller budgets or evaluating specific generation techniques. Understand these distinctions to optimize your synthetic data strategy.

Key insights

Fixed-Source Synthesis (FSS) is a bounded synthetic data scaling method, distinct from Source Expansion, with performance limits and specific characteristics.

Principles

• Small budgets: Source Expansion (SE) and Fixed-Source Synthesis (FSS) are comparable.
• Large budgets: SE (adding seeds) outperforms FSS (more responses).
• Fixed-Source Synthesis (FSS) offers bounded scaling.

Method

Isolate Fixed-Source Synthesis (FSS) by fixing seed-question pool and teacher model, varying per-question response budget via Rejection Sampling. Adapt rectified scaling law to model performance.

In practice

• Use FSS to compare synthesis protocols.
• For large-scale data, prioritize Source Expansion.
• Plain Rejection Sampling is effective within FSS.

Topics

• Synthetic Data
• Data Scaling
• Source Expansion
• Fixed-Source Synthesis
• Rejection Sampling
• Scaling Laws

Best for: Research Scientist, AI Scientist

Related on AIssential

• LuminaSFT: Generating Synthetic Fine-Tuning Data for Small Language Models — Synthetic data generation and regeneration significantly improve small language model performance, especially with task-specific approaches.
• Finite Resources False Discovery Rate Control in Structured Hypothesis Spaces — A framework unifies FDR control for finite-data p-value uncertainty and structured hypothesis spaces, improving power and resource allocation.
• Synthetic Benchmarks Overstate Forward-Forward Scaling: Real-Data Limits of Layer-Local Training — Forward-Forward learning's real-data scaling and memory efficiency are significantly overstated by synthetic benchmarks, revealing limitations compared to backpropagation.
• How are you handling training data when public datasets don't match your use case? [D] — Public datasets often fail to meet specific ML project needs, necessitating alternative data sourcing and generation strategies.
• SFT-GRPO Data Overlap as a Post-Training Hyperparameter for Autoformalization — Minimizing SFT-GRPO data overlap significantly improves autoformalization model performance and semantic accuracy.
• Evaluating the role of pretraining dataset size and diversity on single-cell foundation model performance — Single-cell foundation models do not show clear data scaling laws, unlike large language models.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.