2.5-D Decomposition for LLM-Based Spatial Construction

2024-07-18 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, long

Summary

A neuro-symbolic pipeline employing "2.5-D decomposition" significantly improves the accuracy of large language models (LLMs) in spatial construction tasks. This method allows the LLM to plan only in the two-dimensional horizontal plane, while a deterministic executor handles all vertical placements based on column occupancy, effectively eliminating a class of common coordinate errors. On the Build What I Mean (BWIM) benchmark (160 rounds), GPT-4o-mini with this pipeline achieved 94.6% mean structural accuracy, outperforming GPT-4o at 90.3% and the best competing system at 76.3%. An ablation study confirmed 2.5-D decomposition as the dominant contributor, accounting for a 50.7 percentage point accuracy increase. The pipeline also transferred to edge hardware, with Nemotron-3 120B on an NVIDIA Jetson Thor AGX matching cloud results at 94.5% accuracy without prompt modifications. The approach's generalizability was further demonstrated on 500 IGLU collaborative building tasks.

Key takeaway

For Research Scientists developing autonomous construction or assembly systems, you should consider implementing a 2.5-D decomposition strategy. By offloading deterministically computable spatial dimensions (like vertical placement under gravity) from the LLM to a dedicated executor, you can significantly enhance accuracy and reliability, even with smaller, more cost-effective models like GPT-4o-mini or local edge hardware, while reducing systematic coordinate errors.

Key insights

Decomposing LLM spatial tasks by offloading deterministic dimensions to an executor drastically improves accuracy.

Principles

• Remove deterministic dimensions from LLM output space.
• Physical constraints can define deterministic dimensions.
• Systematic LLM errors can be corrected with peephole optimization.

Method

The pipeline involves an LLM planning in 2D (x,z) for block placement, while a deterministic executor computes the vertical (y) coordinate based on column occupancy, along with underspecification handling and peephole prompt optimization.

In practice

• Use 2.5-D decomposition for gravity-constrained construction.
• Implement peephole optimization for systematic LLM errors.
• Prioritize color over count for clarification questions.

Topics

• 2.5-D Decomposition
• LLM Spatial Reasoning
• Neuro-Symbolic Architecture
• Build What I Mean Benchmark
• Peephole Prompt Optimization

Code references

• paulwhitten/AgentWhetters-bwim
• ltl-uva/build_what_i_mean
• hisandan/build-it-3

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• Anthropic's Big AI Design Change: The RED Pill — Deterministic scaffolding, not LLM intelligence, achieves high accuracy in analytical tasks, challenging the value proposition of expensive frontier models.
• Sample-Efficient Post-Training for LEGO Spatial-Physics Reasoning — Physical validity alone is insufficient for reliable LEGO assembly reasoning; semantic and geometric fidelity are crucial.
• Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption — Multi-model LLM schedulers must account for non-linear offloading sensitivity and model reload-dominated preemption costs.
• Generalization in LLM Problem Solving: The Case of the Shortest Path — LLMs generalize spatially but fail at length scaling in shortest-path problems due to recursive instability.
• DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoning — DecompSR is a provably correct dataset for fine-grained analysis of LLM compositional spatial reasoning.
• TopoBench: Benchmarking LLMs on Hard Topological Reasoning — LLMs struggle with topological reasoning due to difficulty extracting spatial constraints, not inherent reasoning limitations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.