SEGA: Spectral-Energy Guided Attention for Resolution Extrapolation in Diffusion Transformers

2026-05-21 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

SEGA, or Spectral-Energy Guided Attention, is a new training-free method designed to enhance Diffusion Transformers (DiTs) for text-to-image generation at resolutions exceeding their original training range. DiTs typically experience performance degradation when generating higher-resolution images. Current training-free approaches, such as Rotary Position Embeddings (RoPE) extrapolation combined with attention scaling, apply uniform, content-agnostic scaling across RoPE components. This often leads to a compromise between maintaining global image structure and preserving fine details. SEGA addresses this by dynamically scaling attention across RoPE components, adapting to the latent's spatial-frequency structure during each denoising step. This adaptive scaling significantly improves both structural coherence and fine-detail fidelity, demonstrating superior high-resolution synthesis compared to existing state-of-the-art training-free baselines.

Key takeaway

For Machine Learning Engineers developing text-to-image models, if you are struggling with Diffusion Transformers (DiTs) performing poorly at resolutions beyond their training data, consider integrating SEGA. This training-free method dynamically adjusts attention scaling based on spatial-frequency, directly improving both structural coherence and fine-detail fidelity in your high-resolution outputs. You can achieve superior image synthesis without retraining, mitigating common trade-offs in existing extrapolation techniques.

Key insights

SEGA dynamically scales DiT attention based on latent spatial-frequency, improving high-resolution image generation beyond training data.

Principles

• Uniform attention scaling creates structure-detail trade-offs.
• Adaptive scaling improves both global structure and fine detail.
• Latent's spatial-frequency guides dynamic attention adjustments.

Method

SEGA dynamically scales attention across Rotary Position Embeddings (RoPE) components. It adapts scaling based on the latent's spatial-frequency structure at each denoising step during inference.

In practice

• Apply SEGA to improve DiT output at higher resolutions.
• Enhance fine-detail fidelity in text-to-image generation.
• Mitigate performance drops when extrapolating DiT resolutions.

Topics

• Diffusion Transformers
• Text-to-Image Generation
• Resolution Extrapolation
• Attention Mechanisms
• Spectral-Energy Guidance
• Computer Vision

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Training-free sparse attention based on cumulative energy filtering — Dynamic thresholding for sparse attention simultaneously optimizes sparsity and accuracy in Diffusion Transformers for video generation.
• RiT: Vanilla Diffusion Transformers Suffice in Representation Space — DINOv2's favorable geometric properties enable vanilla Diffusion Transformers with "x"-prediction to achieve superior, efficient image generation.
• Real-Time AttentionBender: Granular Interactive Network Bending of Video Diffusion Transformers — Real-Time AttentionBender offers granular, interactive control over Video Diffusion Transformers, enhancing creative agency and model understanding.
• Show the Signal, Hide the Noise: Spectral Forcing for Pixel-Space Diffusion — Spectral Forcing improves pixel-space diffusion by explicitly filtering high-frequency noise, optimizing model capacity for signal processing.
• Beyond Fully Random Masking: Attention-Guided Denoising and Optimization for Diffusion Language Models — Attention-guided denoising and optimization (AGDO) significantly enhances dLLM reasoning by leveraging intrinsic token dependencies.
• Building Generative Image & Video models at Scale - Sander Dieleman (Veo and Nano Banana) — Diffusion models generate high-quality audiovisual data through iterative denoising of learned compressed latent representations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.