CoStream: Composing Simple Behaviors for Generalizable Complex Manipulation

2026-06-24 · Source: Artificial Intelligence · Field: Technology & Digital — Robotics & Autonomous Systems, Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

CoStream is a novel framework designed for long-horizon, contact-rich complex manipulation tasks, such as seating a GPU into a PCIe slot, addressing the limitations of existing robotic paradigms. Traditional methods offer high precision but lack generalization, while monolithic end-to-end policies generalize better but struggle with precision on out-of-distribution tasks. CoStream overcomes this by orchestrating foundation models and diverse sensing modalities into three composable core behaviors. These include a semantic behavior for extracting spatial constraints, a predictive behavior for forecasting trajectories via keypoint tracking in imagined videos, and a reactive behavior for high-frequency tactile and force corrections. These outputs compose on a shared SE(3) interface into a single pose command. The framework demonstrated strong gains on 8 real-world tasks, particularly in contact-rich assembly and object transfer, and showed robust recovery from manual perturbations during execution.

Key takeaway

For robotics engineers developing systems for high-precision, contact-rich assembly, CoStream offers a robust approach. You should consider adopting a composable behavior framework to achieve both generalization and precision, avoiding the limitations of monolithic policies or rigid pipelines. This method allows your systems to recover from perturbations and perform complex tasks like GPU seating more reliably.

Key insights

Complex manipulation emerges from composing simple, independent behaviors using foundation models and diverse sensing.

Principles

• Decompose complex tasks into simple behaviors.
• Combine foundation models with diverse sensing.
• Use a shared interface for behavior composition.

Method

CoStream orchestrates semantic, predictive, and reactive behaviors. Semantic extracts constraints, predictive forecasts trajectories, and reactive provides tactile corrections. These compose via right-multiplication on an SE(3) interface.

In practice

• Automate contact-rich assembly tasks.
• Improve object transfer precision.
• Enhance robotic recovery from perturbations.

Topics

• Robotic Manipulation
• Behavior Composition
• Foundation Models
• Tactile Sensing
• Precision Assembly
• SE(3) Control

Best for: Research Scientist, AI Scientist, Robotics Engineer

Related on AIssential

• RoboDream: Compositional World Models for Scalable Robot Data Synthesis — A generalizable embodiment-centric world model synthesizes photorealistic robot demonstrations by decoupling motion from environment synthesis.
• WatchAct: A Benchmark for Behavior-Grounded Robot Manipulation — Robot manipulation benchmarks must incorporate observed human behavior to evaluate complex reasoning beyond single-image instructions.
• ATOM-Bench: A Real-World Benchmark for Atomic Skills and Compositional Generalization in Manipulation Policies — Current robotic manipulation policies struggle with fine motor skills and compositional generalization, even when atomic skills are strong.
• Towards Generalizable Robotic Manipulation in Dynamic Environments — Dynamic manipulation in robotics requires specialized datasets and architectures for effective spatiotemporal reasoning.
• Mana: Dexterous Manipulation of Articulated Tools — Dexterous manipulation of articulated tools can be effectively modeled and solved as an animation problem using a sim-to-real framework.
• Geometric Action Model for Robot Policy Learning — GAM integrates 3D geometric reasoning into robot policy learning by repurposing a pretrained geometric foundation model for perception, prediction, and action.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.