Bounding Boxes as Goals: Language-Conditioned Grasping via Neuro-Symbolic Planning

2026-06-12 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Robotics & Autonomous Systems, Artificial Intelligence & Machine Learning · Depth: Expert, long

Summary

GRASP is a lightweight neuro-symbolic framework designed for open-vocabulary tabletop manipulation, translating natural-language queries into neuro-symbolic goal states grounded via a bounding-box detection pipeline. This approach, which uses a pretrained Vision-Language Model (GroundingDINO) and an LLM (GPT-5.2), enables robots to interpret abstract spatial concepts like "top shelf" and execute tasks without additional fine-tuning or extensive training. Experiments on a differential claw arm across 90 trials and three difficulty levels (easy, medium, hard) yielded overall success rates of 86.67%, 76.67%, and 56.67% respectively, averaging 73.33%. Ablation studies confirmed the critical role of closed-loop control, smoothing and deadband, and highest-logit target selection for reliable grasping performance.

Key takeaway

For AI Engineers developing robotic manipulation systems, GRASP offers a lightweight, training-free approach to integrate natural language commands. You should consider adopting neuro-symbolic planning with pretrained VLMs like GroundingDINO to achieve zero-shot generalization and interpret abstract spatial concepts. This framework reduces computational overhead and training data requirements, enabling faster deployment of robust, language-conditioned robots in dynamic environments.

Key insights

GRASP combines VLMs and symbolic planning for zero-shot, language-conditioned robotic manipulation using bounding box goals and closed-loop control.

Principles

• Decouple high-level reasoning from low-level control.
• Closed-loop feedback is critical for robust grasping.
• Neuro-symbolic approaches enhance interpretability and efficiency.

Method

GRASP uses GPT-5.2 for goal state JSON generation and GroundingDINO for real-time bounding box detection. A proportional RPY controller, with smoothing and deadband, aligns the claw based on object-to-center distance.

In practice

• Use GroundingDINO for open-vocabulary object detection.
• Implement proportional RPY control for precise adjustments.
• Employ smoothing and deadband to stabilize robot movements.

Topics

• Neuro-Symbolic AI
• Robotic Manipulation
• Vision-Language Models
• GroundingDINO
• Zero-Shot Learning
• Proportional Control

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Robotics Engineer, AI Engineer

Related on AIssential

• Bounding Boxes as Goals: Language-Conditioned Grasping via Neuro-Symbolic Planning — GRASP enables robots to interpret natural language and perform open-vocabulary grasping using VLMs and bounding boxes, achieving 73.3% success without fine-tuning.
• Grounding vs. Compositionality: On the Non-Complementarity of Reasoning in Neuro-Symbolic Systems — Symbol grounding alone is insufficient for compositional generalization; explicit reasoning objectives are required.
• Differentiable Learning of Lifted Action Schemas for Classical Planning — A novel neural network learns lifted action schemas from state traces with unobserved action arguments.
• CLASP: Language-Driven Robot Skill Selection and Composition using Task-Parameterized Learning — CLASP integrates TP-KMPs with VLMs for data-efficient, language-driven robot skill learning, selection, and composition.
• GroundedPlanBench: Spatially grounded long-horizon task planning for robot manipulation — Jointly planning actions and spatial grounding improves robot task success and action accuracy for VLMs.
• Bridging the Morphology Gap: Adapting VLA Models to Dexterous Manipulation via Intent-Conditioned Fine-Tuning — Adapting VLA models to dexterous hands is possible by decoupling control via a virtual grasp intent and a two-stage learning architecture.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.