Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulation

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

Point as Skeleton is a generative sensor simulation framework designed for end-to-end autonomous driving (E2E-AD) evaluation. It addresses the trade-off between closed-loop interactivity, exemplified by CARLA, and real-world visual fidelity, as seen in nuScenes, by synthesizing visual observations autoregressively. The system utilizes step-wise updated ego states, actor states, scene maps, and point-cloud skeleton conditions. Key innovations include "Reset-and-Roll" for adapting rolling diffusion inference to prevent future-conditioned latent states from committing across simulation steps, and point-cloud skeletons that decouple foreground and background assets into camera-view painted-point and template-depth conditions. Experiments on nuScenes and nuPlan datasets demonstrate improved autoregressive generation quality and visual fidelity during closed-loop rollout, with the code available on GitHub.

Key takeaway

For AI Scientists and Machine Learning Engineers developing end-to-end autonomous driving systems, this work offers a robust approach to high-fidelity closed-loop simulation. You should consider integrating point-cloud-conditioned autoregressive generative models to overcome visual fidelity and error accumulation challenges. This enables more realistic evaluation of policy-induced actions and off-log trajectories, improving model robustness before real-world deployment.

Key insights

Point as Skeleton enhances closed-loop autonomous driving simulation by stabilizing autoregressive generation with point-cloud conditions.

Principles

• Decouple foreground and background assets.
• Prevent future-conditioned latent state commitment.
• Use point-cloud skeletons for geometric cues.

Method

The method involves "Reset-and-Roll" autoregressive generation, which caches current latents before using lookahead layouts, and "Point Cloud Skeleton" conditions (color map, template-based depth map) for view-consistent geometry.

In practice

• Implement Reset-and-Roll for stable AR generation.
• Utilize point-cloud skeletons for scene representation.
• Augment foreground instances during training.

Topics

• Autonomous Driving Simulation
• Generative Sensor Simulation
• Point Cloud Skeletons
• Autoregressive Video Generation
• Closed-Loop Evaluation
• Diffusion Models

Code references

• krauwu/point-as-skeleton

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• Safe Autoregressive Image Generation with Iterative Self-Improving Codebooks — Unified multimodal models can self-identify and eliminate unsafe image generations through iterative codebook refinement.
• PTL-Diffusion: Manifold-Aware Diffusion with Periodic Terminal Laws — PTL-Diffusion improves manifold data generation by using periodic, structured terminal laws in its forward noising process.
• Future Forcing: Future-aware Training-free KV Cache Policy for Autoregressive Video Generation — Future Forcing uses stable pre-RoPE query distributions to enable training-free, future-aware KV cache management for AR video generation.
• T2LM: Long-Term 3D Human Motion Generation from Multiple Sentences — T2LM generates smooth, long-term 3D human motion from raw text without needing long-term training data or post-processing.
• A Generative Model for Closed-Loop Microsimulation of Signalized Intersections — Actor-centric generative model Enactor enables stable, realistic closed-loop microsimulation of signalized intersections by learning from its own predictions.
• CLEAR: Cognition and Latent Evaluation for Adaptive Routing in End-to-End Autonomous Driving — CLEAR achieves real-time, multi-modal autonomous driving by replacing iterative denoising with single-step latent space drift and semantic reasoning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.