Scale Up Strategically: Learning Compositional Generalization via Bias-Aware Evaluation and Data Collection for Robotic Manipulation

2026-07-23 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Robotics & Autonomous Systems, Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

A new diagnostic framework and bias-aware data collection strategy addresses compositional generalization failures in robotic manipulation policies. This framework localizes policy shortcuts to individual "instruction factors" like color, verb, object, size, and spatial attribute, which are reusable semantic components. It formalizes "instruction factor bias"—the tendency of fine-tuned policies to over-rely on dominant factors—and quantifies it using Factor Dominance Rate (FDR) and Factor Dominance Hierarchy (FDH). Evaluation across six foundation policies revealed a consistent ordering: color ≥ object ≥ spatial ≥ verb ≥ size, with color being dominant and verb/size under-grounded. The proposed bias-aware data collection strategy reallocates a fixed budget towards these under-grounded factors, outperforming baselines in simulation and on a real robot using half the demonstrations, enabling more sample-efficient and generalizable policy learning.

Key takeaway

For Robotics Engineers developing policies for diverse instructions, understanding and mitigating instruction factor bias is crucial. You should diagnose your pretrained policies using metrics like Factor Dominance Rate to identify over-relied-upon semantic components. By strategically reallocating your data collection budget towards under-grounded factors, you can achieve more sample-efficient and generalizable policy learning, potentially halving demonstration requirements and improving real-world robot performance.

Key insights

Instruction factor bias, where policies over-rely on dominant semantic components, hinders compositional generalization in robotic manipulation.

Principles

• Pretrained policies take shortcuts, deferring to salient cues.
• Color, object, and spatial attributes tend to dominate over verb and size.
• Bias-aware data collection improves sample efficiency.

Method

A diagnostic framework quantifies instruction factor bias using Factor Dominance Rate (FDR) and Factor Dominance Hierarchy (FDH) to identify under-grounded factors, guiding bias-aware data collection.

In practice

• Evaluate policies for factor dominance using FDR and FDH.
• Reallocate data budget towards under-grounded instruction factors.

Topics

• Robotics
• Compositional Generalization
• Robotic Manipulation
• Instruction Factor Bias
• Data Collection Strategy
• Policy Learning

Best for: Research Scientist, AI Scientist, Robotics Engineer, Machine Learning Engineer

Related on AIssential

• ATOM-Bench: A Real-World Benchmark for Atomic Skills and Compositional Generalization in Manipulation Policies — Current robotic manipulation policies struggle with fine motor skills and compositional generalization, even when atomic skills are strong.
• Factored Diffusion Policies:Compositionally Generalized Robot Control with a Single Score Network — Factored Diffusion Policies use a single diffusion network with additive score decomposition for compositionally generalized robot control.
• Is Diversity All You Need for Scalable Robotic Manipulation? — Not all data diversity benefits robotic manipulation equally; task diversity and debiased expert data are key.
• See Less, Specify More: Visual Evidence Budgets for Generalizable VLAs — VLA generalization improves by training executors with explicit local guidance and task-sufficient visual evidence, avoiding weak supervision.
• Physical Intelligence, a hot robotics startup, says its new robot brain can figure out tasks it was never taught — The π0.7 model demonstrates compositional generalization in robotics, enabling robots to perform novel tasks by combining learned skills.
• Training and Evaluating Diffusion Policies with Long Context Lengths — Naive context scaling for Diffusion Policies is viable with proper architecture and data, improved by variable history training.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.