LiTo: Surface Light Field Tokenization

2026-03-12 · Source: Apple Machine Learning Research · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Advanced, quick

Summary

LiTo introduces a 3D latent representation that jointly models object geometry and view-dependent appearance, addressing limitations of prior works that focused on either geometry or view-independent diffuse appearance. The approach encodes random subsamples of a surface light field from RGB-depth images into a compact set of latent vectors. This unified 3D latent space representation effectively reproduces view-dependent effects such as specular highlights and Fresnel reflections under complex lighting conditions. Furthermore, the model trains a latent flow matching model on this representation, enabling the generation of 3D objects with appearances consistent with the input image's lighting and materials. Experiments demonstrate that LiTo achieves higher visual quality and better input fidelity compared to existing methods.

Key takeaway

For AI Scientists developing 3D reconstruction and generation systems, LiTo offers a robust method for capturing complex view-dependent appearance. You should consider integrating surface light field tokenization to improve visual fidelity and consistency in generated 3D assets, especially when realistic lighting and material interactions are critical for your applications.

Key insights

LiTo unifies 3D geometry and view-dependent appearance modeling using surface light field tokenization.

Principles

• Surface light fields capture view-dependent effects.
• Latent vectors can compactly represent geometry and appearance.

Method

Encodes random subsamples of RGB-depth surface light fields into latent vectors, then trains a latent flow matching model for 3D object generation.

In practice

• Generate 3D objects from single input images.
• Reproduce realistic specular highlights and Fresnel reflections.

Topics

• Surface Light Fields
• 3D Object Generation
• View-Dependent Rendering
• Latent Representations
• Neural Radiance Fields

Best for: AI Scientist, AI Researcher, Computer Vision Engineer, Research Scientist

Related on AIssential

• 🍓Surface Light Tokenizer🍓 👉Apple unveils LITO a novel latent flow matching model enables... — LITO uses latent flow matching to convert images into high-quality 3D models via compact surface light field representations.
• TokenLight: Precise Lighting Control in Images using Attribute Tokens — Attribute tokens enable precise, continuous control over multiple illumination factors in image relighting.
• TextHOI-3D: Text-to-3D Hand-Object Interaction via Discrete Multi-View Generation and Joint Mesh Optimization — TextHOI-3D uses discrete multi-view visual tokens to bridge text-conditioned generation and geometry-aware 3D hand-object mesh recovery.
• Vision-Reasoning-Guided Occlusion Removal from Light Fields — Combining light field integration with vision-language models robustly removes occlusions for enhanced scene recovery.
• Intrinsic decomposition and editing of 3D Gaussian splats — Intrinsic decomposition for 3D Gaussian splats enables texture editing independent of lighting across viewpoints.
• LUCID: Learning Unified Control for Image Deflaring and Exposure Mastery in Nighttime Photography — LUCID unifies controllable flare mitigation and exposure adjustment for superior nighttime image restoration.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Apple Machine Learning Research.