Unified Safe In-context Image Generation in Multimodal Diffusion Transformers via Restricting Unsafe Information Flows

2026-06-05 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Expert, quick

Summary

Unified Visual Safety Regulator (UVR) is a novel, training-free framework designed to enhance safety in multimodal Diffusion Transformers (DiTs) for image generation, particularly addressing limitations of existing mechanisms in image-to-image (I2I) editing tasks. Current safety solutions are often tailored for text-to-image (T2I) synthesis or U-Net architectures, proving ineffective for DiT-based systems. UVR operates by analyzing attention dynamics and information flow within multimodal attention (MM-Attn), identifying a critical "task-independent start-up stage" where unsafe semantics first appear. It then applies unified, targeted attention modulation and explicitly restricts harmful information flow over these identified unsafe output patches. Experimental results demonstrate UVR's state-of-the-art safety performance, achieving 91% and 77% erase rates in image synthesis and editing tasks, respectively, while maintaining visual quality.

Key takeaway

For Machine Learning Engineers developing multimodal Diffusion Transformers, UVR offers a training-free solution to mitigate unsafe content generation, especially in image-to-image editing. You should consider integrating UVR to achieve high erase rates (91% in synthesis, 77% in editing) without significant visual quality degradation. This approach provides a unified safety mechanism where existing T2I or U-Net-based methods fall short.

Key insights

UVR restricts unsafe information flow in DiT multimodal attention during an early "start-up stage" to prevent harmful image generation.

Principles

• Unsafe semantics emerge early in DiT attention dynamics.
• Harmful signals propagate and entangle with benign content.
• Targeted attention modulation can restrict unsafe flow.

Method

UVR analyzes MM-Attn information flow to identify unsafe output patches during a "start-up stage," then applies unified, targeted attention modulation to explicitly restrict harmful information flow.

In practice

• Apply UVR to DiT-based image synthesis.
• Use UVR for safer I2I editing tasks.
• Integrate UVR into existing DiT frameworks.

Topics

• Diffusion Transformers
• Multimodal Attention
• Image Generation Safety
• Image-to-Image Editing
• Unsafe Content Mitigation
• Attention Modulation

Code references

• deng12yx/UVR

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• Unified Safe In-context Image Generation in Multimodal Diffusion Transformers via Restricting Unsafe Information Flows — UVR unifies safety in multimodal DiTs by modulating attention to restrict unsafe information flow at early, localized stages.
• Robust and Generalizable Safety Steering for Text-to-Image Diffusion Transformers — SafeDIG enables robust and generalizable safety steering for Diffusion Transformers by transferring sparse features across risk domains.
• Safety-Guided Flow (SGF): A Unified Framework for Negative Guidance in Safe Generation — Early, strong negative guidance within a critical time window is crucial for safe and high-quality generative model outputs.
• Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images — Implicit visual exposure to threat-related content can shape safety-oriented personas in MLLMs without explicit safety labels.
• RefDecoder: Enhancing Visual Generation with Conditional Video Decoding — Conditional video decoders improve detail preservation and consistency in visual generation.
• Breaking the Lock-in: Diversifying Text-to-Image Generation via Representation Modulation — Early convergence of the zero-frequency spatial average (DC) component in Transformer features limits text-to-image generation diversity.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.