Text-to-Image Models Keep Getting Better. Their Training Data Keeps Getting Worse.

2026-07-27 · Source: AI on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, quick

Summary

A CVPR paper reveals a counterintuitive trend: newer text-to-image models, despite improved perceptual quality and text-image alignment scores like GenEval and CLIPScore, generate synthetic training data that is less effective for downstream classifiers. Researchers evaluated thirteen open-source models from 2022-2025, generating 100,000 synthetic images for 200 ImageNet classes. Classifiers trained solely on this synthetic data performed progressively worse on real images as the generative model's release date became more recent, falling significantly below the 0.73 accuracy achieved by a ResNet-50 trained on real data. The degradation stems from three issues: systematic texture degradation (while structure survives), poor transfer of high-frequency details, and mode collapse in newer models, leading to high density but low coverage in the synthetic data distribution. A partial remedy using GPT-generated detailed captions improved some aspects but did not fix texture or high-frequency gaps.

Key takeaway

For machine learning engineers evaluating text-to-image models for synthetic data generation, you should prioritize diversity and distributional coverage over mere perceptual quality. Your evaluation metrics must include density-coverage scores and train-on-synthetic, test-on-real accuracy, not just GenEval or CLIPScore. This ensures your synthetic datasets are genuinely useful for training robust downstream models, preventing mode collapse and improving real-world generalization.

Key insights

Newer text-to-image models produce aesthetically pleasing but distributionally narrow synthetic data, hindering downstream classifier performance on real images.

Principles

• Perceptual quality does not equate to data utility.
• Synthetic texture degrades more than structure.
• Mode collapse reduces data diversity.

Method

The paper's method involved generating synthetic ImageNet subsets from 13 models, training ResNet-50 classifiers on these, and testing their accuracy on real images to assess data utility.

In practice

• Evaluate generative models with density-coverage metrics.
• Use train-on-synthetic, test-on-real accuracy.
• Employ detailed prompts for better synthetic data.

Topics

• Text-to-Image Models
• Synthetic Data
• Classifier Training
• ImageNet
• Mode Collapse
• Data Diversity
• Evaluation Metrics

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• The Day Synthetic Data Turned Poisonous: Inside Model Collapse — Recursive training on synthetic data causes model collapse, eroding diversity and amplifying errors.
• Lens: Rethinking Training Efficiency for Foundational Text-to-Image Models — Training efficiency in T2I models is driven by compact size, data density, and convergence speed.
• Does Synthetic Layered Design Data Benefit Layered Design Decomposition? — Purely synthetic layered data can effectively improve graphic design decomposition and overcome real-world data limitations.
• A Zipfian Analysis of Visual Token Distributions for AI-Generated Images — AI-generated images statistically diverge from natural visuals, especially with complex prompts, revealing model-specific "fingerprints."
• The Sequence Knowledge # 780: Synthetic Data for Image Models — Synthetic data generation is crucial for robust vision systems, overcoming real-world data limitations through controlled pixel creation.
• Breaking the Lock-in: Diversifying Text-to-Image Generation via Representation Modulation — Early convergence of the zero-frequency spatial average (DC) component in Transformer features limits text-to-image generation diversity.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI on Medium.