HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures

2026-07-02 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

HERMES introduces a multi-granularity labeling substrate designed to overcome limitations of fixed-granularity data partitioning in pre-training data mixtures. Traditional methods commit to a single semantic axis and granularity, necessitating label rebuilding for resolution changes. HERMES employs a Learned Semantic Transform followed by 3-stage residual vector quantization, annotating each document with a coarse-to-fine code where prefix length dictates granularity, supporting up to approximately 130k cells. While its coarse-granularity performance aligns with KMeans-family methods, its core contribution is the hierarchical substrate itself. During 1B-parameter, 25B-token pre-training, HERMES revealed a nuanced interaction: a specific Stage-2 rule improved a 16-task capability macro-average by +0.0253 at one prefix length, but this advantage vanished at a finer level where candidate pools contracted by approximately 5x. This system reframes data mixture design, allowing navigation of a reusable, data-derived granularity hierarchy instead of selecting from fixed label sets.

Key takeaway

For Machine Learning Engineers designing large-scale pre-training data mixtures, HERMES suggests rethinking your approach to data labeling. Instead of committing to fixed label sets, you should consider implementing hierarchical, data-derived granularity systems. This allows you to dynamically explore how different data mixture rules perform across varying resolutions, potentially uncovering performance gains like the +0.0253 macro-average lift observed, or identifying when specific rules lose their effectiveness as data pools contract.

Key insights

HERMES provides a hierarchical, data-derived labeling system for pre-training data, enabling flexible granularity control.

Principles

• Fixed-granularity labels bottleneck data mixing.
• Hierarchical labeling reveals nuanced data interactions.
• Granularity choice impacts rule efficacy.

Method

HERMES uses a Learned Semantic Transform and 3-stage residual vector quantization to annotate documents into coarse-to-fine codes, with prefix length controlling granularity up to ~130k cells.

In practice

• Explore data mixture rules across granularities.
• Design pre-training pipelines with flexible data views.
• Test rule efficacy at different data resolutions.

Topics

• Data Labeling
• Pre-training Data Mixtures
• Hierarchical Clustering
• Vector Quantization
• Large Language Models
• Semantic Transforms

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• HARP: Efficient Data Selection for Finetuning Large Language Models — HARP efficiently selects finetuning data for LLMs by hierarchically evaluating representative subsets and inferring utilities.
• Bulk Labelling and Prodigy — Bulk labeling uses language model embeddings and dimensionality reduction to visually cluster and rapidly pre-label text subsets for annotation.
• DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts — DAG-MoE improves Mixture-of-Experts performance by replacing weighted-summation with learned structural aggregation, expanding expert-combination space and enabling multi-step reasoning.
• Characterize Then Distill: Mechanistic Reasoning in Large Output Spaces — Reasoning in large output spaces can be mechanistically characterized as a two-phase process: shortlisting followed by fine-grained reasoning.
• RegMix-D: Dynamic Data Mixing via Proxy Training Trajectories — Leveraging full loss trajectories from proxy runs enables dynamic data mixture optimization for LLM pretraining.
• Supervised learning is great — it's data collection that's broken — Supervised learning is effective; data collection and human knowledge reuse are the real problems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.