DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

2026-07-27 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

DataOrchestra is a novel framework designed to optimize Large Language Model (LLM) pretraining data processing by orchestrating example-specific pipelines. Unlike existing methods that apply fixed strategies uniformly, DataOrchestra unifies various processing operations and decides for each data chunk whether to drop, untouch, or clean it. For cleaning, it selects one or more downstream operations, including programmatic editing or LLM-based rewriting, and generates concrete instructions for execution by tool models. Models pretrained from 0.5B to 7B on web data processed by DataOrchestra achieved stable average gains across 11 benchmarks compared to individual data-processing methods. The framework also proved effective for math continued pretraining, outperforming stronger baselines while reducing processing compute by skipping unnecessary operations.

Key takeaway

For ML engineers optimizing LLM pretraining data pipelines, you should consider adopting a dynamic, example-specific curation framework like DataOrchestra. This approach can yield stable performance gains across 11 benchmarks for models from 0.5B to 7B and reduce processing compute compared to uniform strategies, especially for specialized tasks like math pretraining. Evaluate its integration to enhance both model quality and operational efficiency.

Key insights

DataOrchestra dynamically curates pretraining data per-example, unifying operations for improved LLM performance and efficiency.

Principles

• Per-example data processing outperforms fixed strategies.
• Orchestrating diverse operations optimizes data curation.
• Skipping unnecessary steps reduces compute.

Method

DataOrchestra's orchestrator decides to drop, untouch, or clean data chunks. For cleaning, it selects programmatic or LLM-based rewriting operations, generating specific instructions for tool models.

In practice

• Apply dynamic data curation to LLM pretraining.
• Integrate diverse processing tools via an orchestrator.
• Optimize compute by conditional data processing.

Topics

• Pretraining Data Curation
• Large Language Models
• Data Orchestration
• Machine Learning Pipelines
• Computational Efficiency
• Benchmarking

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Reasoning Quality Emerges Early: Data Curation for Reasoning Models — Reasoning quality in LLMs can be effectively improved by curating SFT data based on early token loss patterns, significantly boosting efficiency.
• Optimal Splitting of Language Models from Mixtures to Specialized Domains — Optimizing compute allocation between general pretraining and specialized continued pretraining improves language model performance.
• How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data — Structured pedagogical formats and diverse outputs are key to high-quality synthetic pretraining data, not larger generator models.
• Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality — Web graph centrality effectively guides pretraining data selection for LLMs, capturing complementary knowledge efficiently.
• Data-Efficient Adaptation of Multilingual LLMs to Ukrainian — Data-centric methods enable efficient adaptation of multilingual LLMs to low-resource languages by addressing tokenization, data scarcity, and instruction tuning.
• Building a Local LLM-as-Judge Pipeline for Image Dataset Curation — Combining local LLMs with a Lambda Architecture effectively curates image datasets, improving accuracy and user experience.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.