Optimizing Visual Generative Models via Distribution-wise Rewards

2026-07-02 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

A novel framework addresses limitations in conventional reinforcement learning for visual generative models by employing distribution-wise rewards instead of sample-wise functions. Traditional methods often lead to reward hacking, degrading image diversity and introducing visual anomalies, alongside mode collapse. This new approach finetunes generative models to better align with real-world data distributions by accounting for the overall data distribution of samples, thereby mitigating mode collapse. To overcome the high computational cost of estimating these rewards, the framework introduces a subset-replace strategy, which efficiently provides reward signals by updating only a small subset of a generated reference set. Additionally, it applies RL to optimize post-hoc model merging coefficients, potentially mitigating train-inference inconsistency from stochastic differential equations (SDEs). Experiments demonstrate significant improvements in FID-50K, reducing it from 8.30 to 5.77 for SiT and from 3.74 to 3.52 for EDM2, while also enhancing perceptual quality and preserving sample diversity.

Key takeaway

For machine learning engineers developing visual generative models, if you are encountering issues like mode collapse or degraded image diversity, consider implementing distribution-wise reward functions. This approach, coupled with efficient estimation strategies like subset-replace, can significantly improve your model's FID-50K scores and perceptual quality, as demonstrated by improvements from 8.30 to 5.77 for SiT. Evaluate integrating this method to enhance the realism and diversity of your generated outputs.

Key insights

Distribution-wise rewards and a subset-replace strategy mitigate mode collapse and improve diversity in visual generative models.

Principles

• Sample-wise rewards can cause reward hacking and mode collapse.
• Distribution-wise rewards align models with real-world data distributions.
• Efficient reward estimation is crucial for distribution-wise methods.

Method

Finetune generative models with distribution-wise rewards, using a subset-replace strategy for efficient reward estimation, and optimize post-hoc model merging coefficients via RL.

In practice

• Apply distribution-wise rewards to improve FID-50K in generative models.
• Utilize subset-replace for computationally efficient reward signals.

Topics

• Visual Generative Models
• Reinforcement Learning
• Distribution-wise Rewards
• Mode Collapse Mitigation
• FID-50K
• Image Diversity
• Subset-Replace Strategy

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Better Source, Better Flow: Learning Condition-Dependent Source Distribution for Flow Matching — Optimizing source distribution in flow matching models significantly improves text-to-image generation performance and convergence speed.
• The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation — FID scores are highly variable due to training randomness, making single reported numbers unreliable and requiring new evaluation protocols.
• Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist — Complex T2I prompt faithfulness requires dependency-aware evaluation and a balanced reward system to avoid aesthetic bias.
• From Collapse to Improvement: Statistical Perspectives on the Evolutionary Dynamics of Iterative Training on Contaminated Sources — Model collapse in iterative training can be avoided with fresh data and appropriate sample sizes.
• Drifting Preference Optimization for One-Step Generative Models — DrPO enables efficient preference finetuning for one-step generative models using black-box rewards and feature-space updates.
• Lens: Rethinking Training Efficiency for Foundational Text-to-Image Models — Training efficiency in T2I models is driven by compact size, data density, and convergence speed.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.