UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Software Development & Engineering · Depth: Expert, extended

Summary

UltraX is a novel function-calling refinement framework designed to enhance large-scale pre-training data quality for Large Language Models (LLMs). It addresses diminishing returns from mere data expansion by introducing a complete editing function space, including insertion, deletion, and modification, enabling fine-grained instance-level editing. The framework employs a reliable program-supervision generation pipeline, utilizing dataset-adaptive prompt optimization to guide an expert LLM in producing high-quality refined texts. These are then converted into structured program supervision via Line Alignment Mapping and Dynamic Context Replacement. During inference, UltraX ensures stability and reliability through sliding-window prediction, global operation aggregation, and systematic post-processing. Experiments with 1B models pretrained from scratch on multiple corpora, including FineWeb and RedPajama-V2, demonstrate UltraX achieves the highest average performance, with relative improvements exceeding 2% on several datasets and showing stronger data efficiency with fewer training tokens.

Key takeaway

For Machine Learning Engineers building large language models, prioritizing data quality over raw volume is critical for achieving further performance gains. You should consider adopting programmatic refinement frameworks like UltraX, which consistently improves downstream performance by over 2% with greater data efficiency. This approach, especially effective for low-quality data subsets, minimizes over-editing while preserving valuable content, offering a scalable and reliable method to enhance pre-training data.

Key insights

UltraX programmatically refines large-scale pre-training data, improving LLM performance and data efficiency through fine-grained editing.

Principles

• LLM performance gains now hinge on data quality, not just scale.
• Function-calling refinement offers efficiency and coherence over E2E generation.
• A complete editing function space (insert, delete, modify) is crucial.

Method

UltraX generates expert LLM refined text, maps it to structured program supervision via Line Alignment and Dynamic Context Replacement, trains a lightweight refiner, then executes with sliding-window prediction and post-processing.

In practice

• Implement programmatic editing with insertion, deletion, and modification.
• Use dataset-adaptive prompt optimization for LLM-based refinement.
• Employ sliding-window processing for long document refinement.

Topics

• Large Language Models
• Data Refinement
• Pre-training Data Quality
• Function-Calling Frameworks
• Programmatic Editing
• Data Efficiency

Code references

• openbmb/UltraX
• huggingface/lighteval

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing — UltraX programmatically refines LLM pre-training data at scale, introducing insertion for fine-grained editing to boost quality and efficiency.
• HALO: Hybrid Adaptive Latent Reasoning for Language Models — Selective latent refinement significantly improves frozen LLM performance per unit of compute compared to uniform application.
• CDR-Bench: Evaluating Faithful Execution of Compositional, Order-Sensitive Data Refinement Recipes — Current LLMs struggle with the procedural faithfulness required for compositional, order-sensitive data refinement recipes.
• Database Normalization via Dual-LLM Self-Refinement — Miffie automates database normalization using a dual-LLM self-refinement loop with specialized zero-shot prompts for accuracy and cost-efficiency.
• Reinforced Fast Weights with Next-Sequence Prediction — REFINE improves fast weight models for long-context tasks by shifting from next-token to next-sequence prediction via reinforcement learning.
• DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data — DataOrchestra dynamically curates pretraining data per-example, unifying operations for improved LLM performance and efficiency.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.