SeeThrough3D: Occlusion Aware 3D Control in Text-to-Image Generation

2026-02-26 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

SeeThrough3D is a novel model designed for 3D layout-conditioned text-to-image generation that explicitly addresses occlusion reasoning, a critical challenge in synthesizing partially occluded objects with depth-consistent geometry and scale. Existing methods often struggle with precise inter-object occlusions, leading to unrealistic scene generation. SeeThrough3D introduces an occlusion-aware 3D scene representation (OSCR) where objects are modeled as translucent 3D boxes rendered from a specified camera viewpoint, allowing the model to reason about hidden object regions and provide explicit camera control. It conditions a pretrained flow-based text-to-image model using visual tokens derived from this 3D representation and employs masked self-attention to accurately bind object bounding boxes to their textual descriptions, preventing attribute mixing. The model was trained on a synthetic dataset featuring diverse multi-object scenes with significant inter-object occlusions, demonstrating effective generalization to unseen object categories.

Key takeaway

For research scientists developing 3D layout-conditioned text-to-image models, you should integrate explicit occlusion reasoning into your scene representations. Adopting an approach like SeeThrough3D's OSCR and masked self-attention can significantly improve the realism of inter-object occlusions and prevent attribute mixing, leading to more controllable and geometrically consistent generated images. Consider generating synthetic datasets with diverse occlusion scenarios to robustly train your models.

Key insights

Explicitly modeling occlusions in 3D scene representations enhances text-to-image generation realism and control.

Principles

• Occlusion reasoning is fundamental for 3D-conditioned generation.
• Translucent 3D boxes can encode hidden object regions.
• Masked self-attention prevents object attribute mixing.

Method

SeeThrough3D uses an occlusion-aware 3D scene representation (OSCR) with translucent 3D boxes and rendered viewpoints. It conditions a flow-based text-to-image model with visual tokens and applies masked self-attention for object-text binding.

In practice

• Use OSCR for depth-consistent object synthesis.
• Apply masked self-attention for multi-object scenes.
• Train on synthetic datasets with strong occlusions.

Topics

• Text-to-Image Generation
• Occlusion Reasoning
• 3D Scene Representation
• Masked Self-Attention
• Computer Vision

Best for: Research Scientist, AI Researcher, AI Scientist, Computer Vision Engineer

Related on AIssential

• Occ-VLM: Occupancy Grounded Vision Language Model for Indoor Scene Understanding — Occ-VLM unifies 2D semantics and 3D geometry for scene understanding using only RGB images and occupancy as a spatial prior.
• Stream3D: Sequential Multi-View 3D Generation via Evidential Memory — Stream3D uses an evidential memory and evidence score mechanism to enable consistent 3D generation from sequential multi-view inputs without retraining existing models.
• IV-CoT: Implicit Visual Chain-of-Thought for Structure-Aware Text-to-Image Generation — IV-CoT improves text-to-image generation by separating structural planning from appearance rendering using a latent visual reasoning cascade.
• Direct 3D-Aware Object Insertion via Decomposed Visual Proxies — DIRECT enables 3D-aware object insertion by decomposing guidance into appearance, geometry, and context, injected separately for precise control.
• TextHOI-3D: Text-to-3D Hand-Object Interaction via Discrete Multi-View Generation and Joint Mesh Optimization — TextHOI-3D uses discrete multi-view visual tokens to bridge text-conditioned generation and geometry-aware 3D hand-object mesh recovery.
• PanoPlane: Plane-Aware Panoramic Completion for Sparse-View Indoor 3D Gaussian Splatting — Panoramic scene completion with layout-anchored attention steering significantly improves sparse-view 3D reconstruction.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.