Geometrically Consistent Multi-View Scene Generation from Freehand Sketches

2026-04-15 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

Researchers have introduced a novel framework for generating geometrically consistent multi-view scenes from a single freehand sketch, addressing a previously unattempted problem in computer vision. Freehand sketches present significant challenges due to their abstract nature and inherent spatial distortions. The new approach overcomes these by developing a curated dataset of approximately 9,000 sketch-to-multiview samples, utilizing an automated generation and filtering pipeline. It also incorporates Parallel Camera-Aware Attention Adapters (CA3) to embed geometric inductive biases into a video transformer and employs a Sparse Correspondence Supervision Loss (CSL) derived from Structure-from-Motion reconstructions. This framework synthesizes all views in a single denoising process, eliminating the need for reference images, iterative refinement, or per-scene optimization. The method achieves over 60% improvement in realism (FID) and 23% in geometric consistency (Corr-Acc) compared to two-stage baselines, with up to a 3.7x inference speedup.

Key takeaway

For research scientists developing 3D scene generation models, this work demonstrates a viable path for creating geometrically consistent multi-view outputs from highly abstract inputs like freehand sketches. You should consider integrating camera-aware attention mechanisms and sparse correspondence supervision to enhance both realism and geometric accuracy in your own models, especially when working with challenging, impoverished input data.

Key insights

A new framework generates geometrically consistent multi-view scenes from single freehand sketches using novel data and architectural components.

Principles

• Geometric inductive biases improve multi-view generation.
• Automated data generation can address data scarcity.
• Sparse correspondence supervision enhances consistency.

Method

The method uses Parallel Camera-Aware Attention Adapters (CA3) within a video transformer and a Sparse Correspondence Supervision Loss (CSL) from Structure-from-Motion to synthesize multi-view scenes from sketches in one denoising step.

In practice

• Generate 3D scenes from simple 2D sketches.
• Improve realism and geometric consistency in scene generation.
• Achieve faster inference for multi-view synthesis.

Topics

• Multi-View Scene Generation
• Freehand Sketches
• Camera-Aware Attention Adapters (CA3)
• Sparse Correspondence Supervision Loss
• Geometric Consistency

Best for: Research Scientist, AI Scientist, Computer Vision Engineer, Machine Learning Engineer

Related on AIssential

• View-Consistent 3D Scene Editing via Dual-Path Structural Correspondense and Semantic Continuity — Achieving view-consistent 3D scene editing requires joint distribution modeling across viewpoints.
• TextHOI-3D: Text-to-3D Hand-Object Interaction via Discrete Multi-View Generation and Joint Mesh Optimization — Multi-view visual tokens effectively bridge text-conditioned visual generation and geometry-aware 3D hand-object mesh recovery.
• World Tracing: Generative Pixel-Aligned Geometry Beyond the Visible — World Tracing unifies pixel-aligned depth and generative completion for comprehensive 3D geometry reconstruction.
• Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding — Video generation models inherently learn robust 3D structural priors and physical laws for scene understanding.
• PanoPlane: Plane-Aware Panoramic Completion for Sparse-View Indoor 3D Gaussian Splatting — Panoramic scene completion with layout-anchored attention steering significantly improves sparse-view 3D reconstruction.
• Track2View: 4D-Consistent Camera-Controlled Video Generation via Paired 3D Point Tracks — Track2View uses paired 3D point tracks to enable 4D-consistent, camera-controlled video re-rendering with explicit spatiotemporal correspondences.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.