Picking an Experimentation Platform: A Retrospective

2026-06-06 · Source: Towards Data Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Project & Product Management · Depth: Intermediate, long

Summary

ManyChat faced significant challenges with manual experimentation, including technical headaches for engineers and analysts acting as "human microservices," hindering product velocity. To address this, the company undertook a structured evaluation process to select an experimentation platform, ultimately choosing Eppo. The process involved initial interviews with PMs, analysts, and engineers to convert anecdotal pain points into concrete requirements, followed by a three-layer discovery: desk research, vendor demos, and a Proof of Concept with two finalists. Despite a near-tie in technical scores between Eppo and Statsig, the decision hinged on which platform better supported ManyChat's long-term vision of nudging non-technical users towards conclusive experiments by default, prioritizing PM-facing UX and governance. Post-contract, the focus shifted to change management, establishing governance, process, and developing people's skills, recognizing that the tool itself is only one part of a successful experimentation program.

Key takeaway

For Directors of Product evaluating experimentation platforms, recognize that technical parity among leading vendors means your decision should prioritize organizational fit and long-term vision. Focus on how a platform nudges non-technical users towards conclusive experiments and plan extensively for post-contract change management. Your work truly begins after signing, requiring significant investment in governance, process, and skills development to ensure the tool delivers trustworthy results and velocity.

Key insights

Platform selection needs rigorous process, but organizational vision and post-contract change management are critical for true experimentation success.

Principles

• A credible decision is the deliverable, not the platform.
• Tools provide flow, but governance and process ensure trustworthy results.
• Interviews are crucial for defining pain points and building credibility.

Method

Conduct interviews to define pain points, then use a phased discovery (desk research, demos, POC) to narrow options. Integrate technical evaluation with organizational vision for the final decision.

In practice

• Convert anecdotal pain into concrete vendor requirements.
• Phase platform discovery to de-risk high-stakes decisions.
• Budget for post-contract change management and training.

Topics

• Experimentation Platforms
• A/B Testing
• Product Management
• Vendor Selection
• Data Governance
• Change Management

Best for: Director of AI/ML, Consultant, AI Product Manager

Related on AIssential

• Building An Effective Experimentation Program (Experimentation Program Series: Guide 01) — Building an effective experimentation program requires more than just technical A/B testing knowledge; it demands structured processes and cross-functional collaboration.
• Platform as a Product: Delivering Value While Balancing Competing Priorities — Abby Bangser's talk at GOTO Copenhagen advocates for treating software platforms as products, not just code, to deliver sustainable value by balancing engineering, design, usability, security, and internal customer needs.
• Data products: Closing the gap between technical excellence and business value — Data products bridge the gap between technical data excellence and business value by focusing on user needs and adoption.
• A Need for Nuance: The Economist’s Andrew Palmer — AI excels at rapid prototyping but faces significant challenges in production deployment and organizational integration.
• Data Products Are Not the Destination. They Are What Gets You There. — Data products serve as capability infrastructure, enabling better decisions, trust, speed, and AI readiness, rather than being an end goal.
• Driving Experimentation Forward through a Working Group (Experimentation Program Series: Guide 03) — An experimentation working group, led by a program manager, drives business results through structured, cross-functional collaboration.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.