Affordance-Based Manipulation Planning with Text Goals and Sim-to-Real Generalisation via Real-to-Sim Image Conversion

2026-07-13 · Source: Artificial Intelligence · Field: Technology & Digital — Robotics & Autonomous Systems, Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new manipulation planning system, published on 2026-07-13, integrates affordance recognition and action effect prediction to enable robots to reason through visual futures. This system evaluates candidate plans by matching predicted outcomes with run-time text-based goals using a multi-modal module. A key capability is its ability to track object positions, even when occluded or when initial descriptors fail, allowing for robust action plan generation. Furthermore, the system incorporates an image conversion module that translates real-world state images, featuring varied object shapes and appearances, into a consistent visual format. This real-to-sim conversion significantly facilitates manipulation planning for physical robot setups. The system's performance was evaluated both in isolated modules and as an integrated unit, demonstrating its capabilities on challenging tasks in both simulation and hardware environments.

Key takeaway

For robotics engineers developing manipulation systems, this approach offers a path to enhanced robustness and flexibility. You should consider integrating affordance-based planning with real-to-sim image conversion to improve generalization from simulation to physical hardware. Utilizing text-based goals can simplify task specification and enable dynamic adaptation, even when objects are occluded. This system's ability to track objects through visual predictions provides a blueprint for more resilient real-world robot deployments.

Key insights

The system combines affordance recognition, action effect prediction, and real-to-sim image conversion for robust robot manipulation with text goals.

Principles

• Reasoning through visual futures enhances planning.
• Text-based goals enable flexible task specification.
• Real-to-sim conversion improves sim-to-real generalization.

Method

The system predicts visual futures, evaluates plans against text goals via multi-modal matching, and uses real-to-sim image conversion for physical robot deployment.

In practice

• Manipulate objects with varied appearances.
• Plan actions despite object occlusion.
• Use natural language for robot task goals.

Topics

• Affordance Recognition
• Robot Manipulation
• Sim-to-Real Generalization
• Text-Based Goals
• Action Planning
• Image Conversion

Best for: Research Scientist, Robotics Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• Making Foresight Actionable: Repurposing Representation Alignment in World Action Models — Aligning visual representations with action-grounded semantics improves robot manipulation policies in World Action Models.
• Decoupled Object-Centric Video Understanding for Generating Robotic Manipulation Commands — Decoupling action recognition from object identification improves robotic command generation from video.
• ADAPT: Benchmarking Commonsense Planning under Unspecified Affordance Constraints — Embodied agents require dynamic affordance reasoning to adapt to unspecified, changing real-world object constraints.
• RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies — RoboDojo unifies sim-and-real evaluation to diagnose generalist robot manipulation policies' capabilities and limitations.
• SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulation — SAFE-Pruner uses future-aware attention forecasting and adaptive subtask division to prevent premature critical token removal in VLA models.
• Mana: Dexterous Manipulation of Articulated Tools — Mana reinterprets dexterous manipulation of articulated tools as an animation problem, enabling scalable sim-to-real transfer.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.