CanvasAgent: Enabling Complex Image Creation and Editing via Visual Tool Orchestration

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

CanvasAgent is a novel tool-augmented multimodal agent designed for complex image creation and editing, addressing limitations of single-model approaches for multi-step visual tasks. It orchestrates 11 heterogeneous visual tools, including generation, editing, and OCR, through multi-turn interaction. The system is trained using CanvasCraft, a new large-scale multimodal tool-use dataset comprising 140K fully annotated executable trajectories (CanvasCraft-SFT) and 10K reinforcement learning task specifications (CanvasCraft-RL). CanvasAgent employs a two-stage SFT+GRPO training framework, utilizing a hybrid reward that combines outcome- and process-level signals. Experiments show CanvasAgent (SFT+RL) significantly improves overall reward to 0.821, image-prompt alignment to 0.869, and trajectory quality to 0.849, outperforming SFT-only variants and general-purpose MLLMs in complex visual workflows.

Key takeaway

For AI Engineers developing advanced image creation agents, recognize that single-model solutions are inadequate for complex, multi-step visual tasks. You should prioritize designing systems that orchestrate heterogeneous tools and manage intermediate visual assets explicitly. Implement a two-stage SFT+RL training framework with a hybrid reward to optimize both final image quality and robust tool-use trajectories. This approach enables more adaptive and effective visual manipulation.

Key insights

Complex image creation benefits from orchestrating diverse visual tools through multi-turn, stateful interaction.

Principles

• Long-horizon visual tasks need stateful tool orchestration.
• Hybrid rewards optimize both output quality and process validity.
• SFT provides a stable foundation for RL-based tool-use.

Method

CanvasAgent trains with two stages: SFT on 140K trajectories, then GRPO on 10K RL tasks using a hybrid reward balancing outcome and process scores. It inspects intermediate results and tracks assets.

In practice

• Decompose complex visual requests into tool trajectories.
• Explicitly manage intermediate visual assets.
• Use perception tools for closed-loop verification.

Topics

• CanvasAgent
• Multimodal Agents
• Image Creation
• Image Editing
• Tool Orchestration
• Reinforcement Learning

Code references

• GML-FMGroup/CanvasAgent

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• CanvasAgent: Enabling Complex Image Creation and Editing via Visual Tool Orchestration — CanvasAgent orchestrates visual tools for complex image creation, trained on CanvasCraft's large-scale executable trajectories.
• VISUALSKILL: Multimodal Skills for Computer-Use Agents — Multimodal skills, integrating visuals with text, significantly improve computer-use agents' performance on GUI tasks.
• Visually-grounded Humanoid Agents — A two-layer paradigm enables digital human agents to perceive, reason, and act autonomously in novel 3D environments using visual input.
• From Plans to Pixels: Learning to Plan and Orchestrate for Open-Ended Image Editing — An experiential framework uses a planner and orchestrator with reward-driven execution for complex, multi-step image editing.
• GPT-Image-2 — Advanced AI models are converging on multimodal capabilities and autonomous agentic workflows, enhancing both creative and technical tasks.
• SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creation — SimWorlds enables dynamic 4D scene generation from text using a multi-agent system, outperforming static approaches.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.