Dynamic Cluster Data Sampling for Efficient and Long-Tail-Aware Vision-Language Pre-training

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

Dynamic Cluster Data Sampling (DynamiCS) is a novel approach for efficient and long-tail-aware Vision-Language Model (VLM) pre-training. It addresses the high computational cost of VLM training and the underrepresentation of rare concepts by dynamically sampling data at each epoch. DynamiCS downsamples large semantic clusters and upsamples small ones, contrasting with methods that merely flatten data distributions. Experiments show DynamiCS reduces VLM training costs, achieving performance competitive with full-scale OpenCLIP while using only about 3% of its training resources. It significantly improves zero-shot classification accuracy on long-tail datasets like Let-it-wag! and ImageNet-1K, outperforming existing cost-saving and dual-purpose baselines on LAION-400M and DataComp-DFN datasets.

Key takeaway

For AI Scientists and Machine Learning Engineers optimizing VLM pre-training, DynamiCS offers a compelling strategy to drastically cut computational costs while boosting performance on long-tail concepts. You should consider integrating dynamic cluster-based sampling, particularly with an α value around 0.2, to achieve significant resource savings (e.g., 3% of full-scale training GPU-hours) without sacrificing, and often improving, model accuracy on diverse and rare categories. This shifts focus from merely flattening data distributions to optimizing for utility.

Key insights

Dynamic cluster-based sampling efficiently trains VLMs, improving long-tail concept performance by balancing data distribution.

Principles

• Dynamic sampling enhances training data diversity.
• Data sampling needs both down- and upsampling.
• "Aim for utility" beats "aim for even" in data balancing.

Method

DynamiCS uses k-means clustering on image embeddings, then applies a scaling factor α (e.g., 0.2) to downsample large clusters and upsample small ones, drawing a different subset each epoch.

In practice

• Cluster image embeddings using DINOv2 and k-means.
• Apply a scaling factor α=0.2 for balanced sampling.
• Dynamically resample clusters in each training epoch.

Topics

• Vision-Language Models
• Data Sampling
• Long-Tail Concepts
• Computational Efficiency
• Cluster Scaling
• Pre-training Optimization

Code references

• MingliangLiang3/DynamiCS
• rom1504/img2dataset

Best for: Research Scientist, AI Engineer, Computer Vision Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• SAMPLe: SAM-based Optimizer for Prompt Learning in VLMs — SAMPLe enhances VLM prompt generalization by optimizing for flatter, lower-loss minima through dynamic sharpness-aware gradient adjustments.
• Synthetic Benchmarks Overstate Forward-Forward Scaling: Real-Data Limits of Layer-Local Training — Forward-Forward learning's real-data scaling and memory efficiency are significantly overstated by synthetic benchmarks, revealing limitations compared to backpropagation.
• Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases — Smaller datasets with repetition accelerate learning by inducing beneficial sampling biases that adjust layer-wise growth.
• HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures — HERMES provides a hierarchical, data-derived labeling system for pre-training data, enabling flexible granularity control.
• GRASS: Gradient-based Adaptive Layer-wise Importance Sampling for Memory-efficient Large Language Model Fine-tuning — GRASS adaptively samples LLM layers for fine-tuning based on gradient norms, significantly reducing memory while boosting performance.
• RegMix-D: Dynamic Data Mixing via Proxy Training Trajectories — Leveraging full loss trajectories from proxy runs enables dynamic data mixture optimization for LLM pretraining.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.