Building An Effective Experimentation Program (Experimentation Program Series: Guide 01)

2022-03-20 · Source: ML in Production · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Project & Product Management · Depth: Intermediate, short

Summary

This content introduces a blog series detailing the creation and implementation of an experimentation program (ExPr) at 2U over 16 months. The author, who manages a team of 5 data scientists and 3 engineers, aims to fill a gap in existing literature by describing the end-to-end process of building such a program from scratch, beyond just the technical aspects of A/B testing. The program has successfully run dozens of experiments, operationalized results leading to substantial increases in operational efficiencies, and developed effective stakeholder interaction models and infrastructure for designing, launching, and analyzing controlled experiments. The series targets data science leadership, including chief data scientists, VPs, senior managers, and data science product managers, but also offers value to individual contributors leading or considering management paths in experimentation.

Key takeaway

For data science leaders and product managers aiming to establish or mature an experimentation program, you should prioritize developing robust cross-functional processes and infrastructure alongside technical A/B testing expertise. Your success hinges on fostering strong relationships with business stakeholders and defining clear responsibilities to move from hypothesis to experimentally-driven conclusions, ultimately driving measurable business outcomes.

Key insights

Building an effective experimentation program requires more than just technical A/B testing knowledge; it demands structured processes and cross-functional collaboration.

Principles

• Iterative improvement drives product enhancement.
• Experimentation optimizes business outcomes.
• Cross-functional collaboration is key for ExPr success.

Method

The author's method involved 16 months of research, development, and iterative improvements to establish processes, infrastructure, and institutional knowledge for running trustworthy online controlled experiments, focusing on small-to-medium sized tests.

In practice

• Focus on operational efficiency experiments.
• Develop internal processes for experiment management.
• Build infrastructure for experiment design and analysis.

Topics

• Experimentation Programs
• A/B Testing
• Data Science Management
• Business Optimization
• Experimentation Infrastructure

Best for: VP of Engineering/Data, AI Product Manager, Data Scientist

Related on AIssential

• Driving Experimentation Forward through a Working Group (Experimentation Program Series: Guide 03) — An experimentation working group, led by a program manager, drives business results through structured, cross-functional collaboration.
• Picking an Experimentation Platform: A Retrospective — Platform selection needs rigorous process, but organizational vision and post-contract change management are critical for true experimentation success.
• Inside Google’s System for Coordinated A/B Testing Across Its Global Service Fleet — Google's system standardizes large-scale A/B testing across distributed services, ensuring statistical rigor and minimizing interference.
• The "Perfect Code" Myth — Embracing iterative development and acknowledging multiple valid solutions fosters confidence in navigating uncertain programming projects.
• Why AI that works in the lab often fails in production — and what actually fixes it — Successful enterprise AI requires bridging foundational research with applied development and rigorous, continuous evaluation.
• Better Experiments with LLM Evals — A funnel, not a fork — LLM evals verify quality before experiments, which validate user outcomes, forming a crucial calibration funnel.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by ML in Production.