FORGE: Towards Functional Tool-Use Generalization via Keypoint Trajectory Reasoning

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

FORGE, a two-stage policy developed by MARS Lab and Georgia Institute of Technology, addresses the challenge of functional generalization in robotic tool-use, where robots must repurpose novel objects for a common function like hitting a nail. Current manipulation policies overfit to specific tools, failing to transfer functions to unseen ones because perceptual similarity does not translate to action space. The proposed FunctiOnal Reasoning and Grounded Execution (FORGE) policy utilizes 2D keypoint trajectories as an intermediate representation, which best balances functional expressiveness and action groundability. This two-stage approach decouples functional reasoning, predicting generalizable keypoint trajectories from action-free data, from action execution, grounding these plans into robot actions with limited demonstrations. On a seven-tool hitting-function benchmark, FORGE consistently outperformed state-of-the-art methods on unseen tools in both simulation and real-world settings, achieving over 2x improvement in average success rate.

Key takeaway

For Robotics Engineers developing robust tool-use policies for novel objects, traditional end-to-end policies struggle with functional generalization. You should consider a two-stage approach like FORGE, leveraging 2D keypoint trajectories as an intermediate representation. This allows you to train functional reasoning on abundant action-free data and ground actions with limited, costly demonstrations, significantly improving success rates on unseen tools.

Key insights

2D keypoint trajectories effectively bridge the perception-to-action gap for robotic functional generalization with novel tools.

Principles

• Functional generalization requires intermediate representations.
• Keypoint trajectories balance expressiveness and groundability.
• Function-aware keypoints are crucial, not generic tracking.

Method

FORGE uses a two-stage policy: first, predict future 2D keypoint trajectories from action-free data using a conditional flow-matching model; then, ground these plans into robot actions with limited action-labeled demonstrations.

In practice

• Use 2D keypoint trajectories for tool-use generalization.
• Decouple functional reasoning from action execution.
• Leverage action-free data for keypoint prediction.

Topics

• Functional Generalization
• Robotic Tool-Use
• Keypoint Trajectories
• FORGE Policy
• Flow-Matching Models
• Manipulation Benchmarks

Best for: Research Scientist, AI Scientist, Robotics Engineer, Machine Learning Engineer

Related on AIssential

• FORGE: Towards Functional Tool-Use Generalization via Keypoint Trajectory Reasoning — Functional tool-use generalization in robots requires decoupling functional reasoning from action execution via keypoint trajectories.
• Beyond Waypoints: A Trajectory-Centric Waypointing Paradigm for Vision-Language Navigation — Grounding waypoints in executable trajectories resolves planning-control inconsistencies and improves reachability in Vision-Language Navigation.
• What Objects Enable, Not What They Are: Functional Latent Spaces for Affordance Reasoning — A4D uses functional latent spaces and uncertainty-aware discovery for generalizable, real-time robot affordance reasoning.
• RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies — RoboDojo unifies sim-and-real evaluation to diagnose generalist robot manipulation policies' capabilities and limitations.
• From Imitation to Alignment: Human-Preference Flow Policies for Long-Horizon Sidewalk Navigation — Bridging imitation learning with human preference alignment significantly enhances robot navigation robustness and social compliance.
• Geometric Action Model for Robot Policy Learning — GAM integrates 3D geometric reasoning into robot policy learning by repurposing a pretrained geometric foundation model for perception, prediction, and action.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.