A Dataset is Worth 1 MB

2026-02-26 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Advanced, quick

Summary

PLADA (Pseudo-Labels as Data) is a novel method designed to significantly reduce communication costs associated with distributing large datasets to multiple clients. It addresses the challenge of transmitting raw data for local model training, especially when pre-trained models are infeasible due to diverse client hardware and software. Unlike dataset distillation, PLADA eliminates pixel transmission entirely by assuming clients possess a large, unlabeled reference dataset like ImageNet-1K or ImageNet-21K. Task knowledge is transferred by sending only class labels for specific images, resulting in payloads under 1 MB. A pruning mechanism filters the reference dataset to retain only semantically relevant images, maximizing training efficiency and minimizing transmission size. Experiments across 10 diverse datasets confirm PLADA's ability to maintain high classification accuracy with minimal data transfer.

Key takeaway

For AI Scientists developing distributed machine learning systems, PLADA offers a compelling approach to drastically cut data distribution costs. You should investigate integrating PLADA's pseudo-label transmission and pruning mechanism, especially when clients have access to large, generic reference datasets, to achieve efficient task knowledge transfer with minimal bandwidth usage and high accuracy.

Key insights

PLADA transfers task knowledge by transmitting only pseudo-labels for a pre-existing reference dataset, eliminating pixel transmission.

Principles

• Utilize pre-loaded reference datasets.
• Transmit labels, not pixels.
• Prune reference data for semantic relevance.

Method

PLADA uses a pruning mechanism to filter a large, unlabeled reference dataset, retaining only labels of semantically relevant images for a target task, which are then transmitted to clients.

In practice

• Reduce data transfer to <1 MB.
• Leverage ImageNet-1K/21K as reference.
• Support diverse client frameworks.

Topics

• Pseudo-Labels as Data
• Dataset Compression
• Data Pruning
• Efficient Data Transfer
• Computer Vision

Best for: AI Scientist, Research Scientist, AI Researcher, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Selecting Samples on Graphs: A Unified Dataset Pruning Framework for Lossless Training Acceleration — A unified graph-based framework prunes datasets by modeling samples as nodes and relations as edges, solving a Maximum Weight Clique Problem.
• Data Labeling Services for AI/ML: Building the Foundation of Intelligent Systems — High-quality labeled data is fundamental for training accurate and reliable AI and machine learning models.
• Transfer Learning in Machine Learning and Deep Learning — Transfer learning reuses pre-trained model knowledge to accelerate new task learning with less data.
• A Review of Pseudo-Labeling for Computer Vision — Pseudo-labeling extends beyond semi-supervised learning to enhance self-supervised and unsupervised methods.
• Cram Less to Fit More: Training Data Pruning Improves Memorization of Facts — Training data pruning and frequency flattening significantly improve LLM factual memorization within capacity limits.
• Can You Build Effective AI Models with Small Datasets? — Effective AI models can be built with small datasets by prioritizing data quality and employing specific technical strategies.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.