TextHOI-3D: Text-to-3D Hand-Object Interaction via Discrete Multi-View Generation and Joint Mesh Optimization

2026-06-10 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Emerging Technologies & Innovation · Depth: Expert, medium

Summary

TextHOI-3D is a novel staged framework designed for text-to-3D hand-object interaction generation, addressing challenges like preserving language semantics, cross-view consistency, object geometry, articulated hand shape, and physically plausible contact. This system employs generated multi-view observations as an explicit interface between text-conditioned visual generation and geometry-aware hand-object recovery. TextHOI-3D learns a compact VQ token space for fixed-camera hand-object observations, then predicts multi-view visual tokens from text using a CLIP-conditioned visual autoregressive model. It recovers a unified hand-object mesh through prior initialization, multi-view joint optimization, and anti-penetration refinement. Evaluations on HO3D-derived datasets show the multi-view setting significantly reduces object CD from 17.26 mm to 4.92 mm and penetration volume from 5.3721 cm^3 to 0.2193 cm^3, while also improving hand errors and surface F-scores.

Key takeaway

For 3D graphics developers or robotics engineers building text-to-3D interaction systems, TextHOI-3D offers a robust approach to generating physically plausible hand-object meshes. You should consider adopting a staged framework that separates semantic generation from geometric recovery, leveraging multi-view visual tokens as an intermediate representation. This method demonstrably improves geometric accuracy and reduces object penetration, crucial for realistic simulations and interactive applications.

Key insights

TextHOI-3D uses discrete multi-view visual tokens to bridge text-conditioned generation and geometry-aware 3D hand-object mesh recovery.

Principles

• Separate semantic generation from geometric recovery.
• Multi-view representations enhance 3D consistency.
• Explicit interfaces improve complex 3D synthesis.

Method

TextHOI-3D learns a VQ token space, predicts multi-view tokens via a CLIP-conditioned autoregressive model, then optimizes a hand-object mesh with prior initialization and anti-penetration refinement.

In practice

• Employ multi-view visual tokens for 3D generation.
• Use anti-penetration refinement for realistic contact.
• Integrate CLIP for robust text-to-visual mapping.

Topics

• Text-to-3D Generation
• Hand-Object Interaction
• Multi-View Synthesis
• Mesh Optimization
• CLIP Model
• VQ-VAE

Code references

• devinli123/MV-SAM3D

Best for: AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Geometrically Consistent Multi-View Scene Generation from Freehand Sketches — A new framework generates geometrically consistent multi-view scenes from single freehand sketches using novel data and architectural components.
• LEXIS: LatEnt ProXimal Interaction Signatures for 3D HOI from an Image — LEXIS-Flow reconstructs 3D human-object interactions from images using dense proximity fields and learned interaction signatures.
• CHOIR: Contact-aware 4D Hand-Object Interaction Reconstruction — CHOIR reconstructs 4D hand-object interactions from monocular video, explicitly using contact as a coupling signal for improved consistency.
• UMI-3D: Extending Universal Manipulation Interface from Vision-Limited to 3D Spatial Perception — UMI-3D enhances robot manipulation data collection via LiDAR-centric SLAM and multimodal sensor fusion.
• Tamaththul3D: High-Fidelity 3D Saudi Sign Language Avatars from Monocular Video — Tamaththul3D enables high-fidelity 3D sign language avatar creation for Arabic Sign Language, outperforming prior methods in speed and accuracy.
• Sketch2Arti: Sketch-based Articulation Modeling of CAD Objects — Sketch2Arti enables intuitive, precise articulation modeling and internal geometry completion for CAD objects using simple 2D user sketches.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.