DeSeG: Decoupling Semantic Intent and Geometric Constraints for Physically Plausible Human-Scene Interaction

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

DeSeG is a novel hierarchical framework designed to synthesize physically plausible human-scene interactions (HSI) by addressing semantic-geometric entanglement and physical hallucinations in generative models. It explicitly decouples high-level semantic intent from low-level geometric constraints. The framework features a Residual Semantic Planner that encodes textual instructions and canonicalized goal voxels into a compact latent space, enabling fine-grained semantic control independent of spatial trajectories. Complementing this, a Physics-Regularized Diffusion Executor integrates differentiable repulsive potential fields directly into its diffusion objective, ensuring collision-aware motion generation during training without test-time optimization. Experiments on the Lingo dataset demonstrate DeSeG's strong performance, reducing mean scene penetration by 47% and improving semantic alignment by 29% compared to existing baselines. It also exhibits robust cross-dataset generalization and superior Semantic-Geometric Consistency on challenging negative-constraint scenarios.

Key takeaway

For AI Scientists and ML Engineers developing human avatar systems or embodied AI, you should recognize that monolithic generative models often conflate semantic intent with geometric cues, leading to physically implausible or semantically incorrect interactions. You can significantly improve the robustness and realism of your human-scene interaction synthesis by adopting hierarchical frameworks like DeSeG. Explicitly decoupling semantic planning from geometric execution, and integrating physics regularization directly into training, will yield more faithful and collision-free motions, even in challenging, counter-intuitive scenarios.

Key insights

DeSeG decouples semantic intent from geometric constraints to synthesize physically plausible human-scene interactions, preventing shortcut learning and penetrations.

Principles

• Semantic-geometric entanglement causes shortcut learning and physical hallucinations in HSI.
• Explicitly decoupling intent from geometry improves robustness and semantic alignment.
• Embedding physics-aware regularization into training ensures efficient collision-free motion.

Method

DeSeG uses a Residual Semantic Planner for latent affordance encoding from text and canonicalized goal voxels, then a Physics-Regularized Diffusion Executor with differentiable repulsive potential fields for collision-aware motion.

In practice

• Canonicalize goal voxels to prevent absolute orientation leakage into latent space.
• Apply joint-specific weights in repulsive potential fields to prioritize critical body parts.
• Warm up physics loss and apply at low noise levels for stable training.

Topics

• Human-Scene Interaction
• Diffusion Models
• Semantic-Geometric Decoupling
• Physical Plausibility
• Motion Synthesis
• Computer Vision

Best for: Research Scientist, AI Scientist, Computer Vision Engineer, Machine Learning Engineer

Related on AIssential

• DeGenseGS: Geometrically and Semantically Decoupled Surgical Scene Understanding in 4D Gaussian Splatting — DeGenseGS decouples semantic and geometric modeling in 4D Gaussian Splatting for robust surgical scene understanding.
• Making Foresight Actionable: Repurposing Representation Alignment in World Action Models — Aligning visual representations with action-grounded semantics improves robot manipulation policies in World Action Models.
• Resolving Representation Ambiguity in Feedforward Novel View Synthesis Transformer via Semantic-Spatial Decoupling — Decoupling semantic and spatial representations in NVS transformers improves rendering fidelity by preventing spatial bias.
• LEXIS: LatEnt ProXimal Interaction Signatures for 3D HOI from an Image — LEXIS-Flow reconstructs 3D human-object interactions from images using dense proximity fields and learned interaction signatures.
• Reinforcing 3D Understanding in Point-VLMs via Geometric Reward Credit Assignment — Targeted reward assignment and geometric consistency verification significantly improve 3D spatial reasoning in Point-VLMs.
• EditSSC: Toward Editable Semantic Occupancy Scenes with Unconditional Diffusion Models — EditSSC repurposes 2D latent diffusion for 3D semantic scene generation, enabling training-free editing capabilities.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.