ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents

2026-05-18 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Emerging Technologies & Innovation · Depth: Expert, extended

Summary

ShopGym is an integrated framework designed for realistic simulation and scalable benchmarking of e-commerce web agents. It addresses the fundamental trade-off between realism and experimental control in existing evaluation methodologies. The framework consists of two main components: ShopArena, which converts live seed storefronts into self-contained, anonymized sandbox shops through a staged generation process, and ShopGuru, which synthesizes benchmark tasks grounded in these simulated storefronts across seven skill categories. ShopGym produces stable, resettable, and inspectable evaluation artifacts that preserve structural properties and agent-evaluation signals relevant to shopping tasks. Validation through graph-based structural analysis and agent-based behavioral evaluation across 224 generated tasks and six sandbox shops (three synthetic, three real-data-based) demonstrates that synthetic shops retain key structural properties and agent performance correlations with live storefronts.

Key takeaway

For research scientists developing and evaluating e-commerce web agents, ShopGym provides a robust solution to the realism-control dilemma. You should consider using this framework to create reproducible and inspectable benchmarks, as it allows for controlled experimentation while preserving the behavioral signals of live storefronts, enabling more reliable comparison and training of next-generation agents.

Key insights

ShopGym offers a scalable framework for creating realistic, controllable, and reproducible e-commerce web agent evaluation environments.

Principles

• Separate storefront exploration from sandbox synthesis.
• Ground synthetic environments in real-world structural data.
• Use multi-agent systems for robust environment generation.

Method

ShopArena explores live storefronts to create anonymized specifications, then synthesizes sandbox shops via a stepwise code generation loop with execution-verification. ShopGuru then generates grounded tasks, including LLM-authored long-horizon journeys, validated by a polish loop.

In practice

• Convert live e-commerce sites into stable sandbox environments.
• Generate diverse, complex shopping tasks for agent evaluation.
• Use LLM-authored tasks with validation loops to ensure feasibility.

Topics

• E-commerce Web Agents
• Realistic Simulation
• Scalable Benchmarking
• ShopGym Framework
• ShopArena

Best for: Research Scientist, AI Scientist, AI Engineer, Machine Learning Engineer

Related on AIssential

• ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents — ShopGym offers a scalable framework for realistic, reproducible e-commerce web agent simulation and benchmarking.
• ClawGym: A Scalable Framework for Building Effective Claw Agents — ClawGym provides a comprehensive framework for developing, training, and evaluating Claw-style personal agents.
• SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agents — SimGym enables rapid, risk-free A/B testing of e-commerce UI changes using VLM agents grounded in real traffic data.
• ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents — Real-world e-commerce user intents are complex, posing significant challenges for current LLM-based agents.
• MM-WebAgent: A Hierarchical Multimodal Web Agent for Webpage Generation — Hierarchical planning and self-reflection improve multimodal AIGC integration for coherent webpage generation.
• Towards Sustainable Growth: A Multi-Value-Aware Retrieval Framework for E-Commerce Search — Balancing immediate conversions with long-term item growth is crucial for e-commerce platform sustainability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.