MolmoWeb: Generating Synthetic Data

2026-03-24 · Source: Ai2 · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Data Science & Analytics · Depth: Intermediate, short

Summary

This content demonstrates how to generate web browsing trajectories using an inference code repository, specifically with the Momo web model and a headless Chromium browser via Playwright. Users define tasks in a JSON format, specifying an ID, prompt, and task type. The system then executes commands to simulate browsing, capturing screenshots, model thoughts, and actions like "go-to" or "mouse click." The output includes a `trajectory.html` visualization, a `trajectory.json` for training, and `metadata.json`. The demonstration also covers using alternative agents, such as the Gemini AX3 agent, which processes an accessibility tree (axe tree) textual representation of web pages instead of screenshots, still producing valid trajectories for training purposes. This approach enables synthetic data generation at scale for pre-training or fine-tuning web agents.

Key takeaway

For AI Engineers building or fine-tuning web agents, this codebase offers a direct method to generate synthetic training data. You can define specific browsing tasks and use either vision-based (Momo web) or accessibility tree-based (Gemini AX3) agents to create realistic interaction trajectories. This capability significantly reduces the need for manual data collection, accelerating model development and iteration cycles for web automation tasks.

Key insights

Generate synthetic web browsing data at scale using LLM or vision-based agents for training.

Principles

• Task definition drives trajectory generation.
• Agents can use visual or accessibility tree inputs.

Method

Define tasks in JSON (ID, prompt, type), execute a command specifying environment and agent (e.g., Momo web, Gemini AX3), then analyze generated `trajectory.html` and `trajectory.json`.

In practice

• Use Playwright for headless browser automation.
• Serve models locally via FastAPI for inference.
• Train web agents on generated `trajectory.json`.

Topics

• Web Agents
• Trajectory Generation
• Synthetic Data Generation
• Large Language Models
• Vision Models

Best for: AI Engineer, Machine Learning Engineer, Data Scientist

Related on AIssential

• State-Grounded Multi-Agent Synthetic Data Generation for Tool-Augmented LLMs — StateGen uses a state-grounded multi-agent LLM loop to generate high-quality synthetic data, eliminating tool-call hallucinations.
• AI2's fully open web agent MolmoWeb navigates the web using only screenshots — MolmoWeb is an open web agent that navigates websites using only visual screenshots, outperforming other open models.
• TrajGenAgent: A Hierarchical LLM Agent for Human Mobility Trajectory Generation — TrajGenAgent generates realistic human mobility trajectories using a hierarchical LLM-agent framework without fine-tuning.
• ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectories — ISE is a three-stage synthesis paradigm for generating execution-grounded, multi-turn OS agent training data.
• LiteCoder-Terminal: Scaling Long-Horizon Terminal Environments for Learning Language Agents — Synthetic, verifiable environments offer a scalable solution for training language agents in complex terminal tasks.
• 「データ不足」の壁を越える：合成ペルソナが日本のAI開発を加速 — Synthetic data can overcome data scarcity and privacy concerns to accelerate AI development, especially in culturally specific domains.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Ai2.