Driving Experimentation Forward through a Working Group (Experimentation Program Series: Guide 03)

2022-05-08 · Source: ML in Production · Field: Business & Management — Operations & Process Management, Project & Product Management, Corporate Strategy & Leadership · Depth: Intermediate, medium

Summary

An experimentation program (ExPr) is a company mechanism using randomized controlled experiments to generate positive business results, comprising people, processes, and infrastructure. This post details how an "experimentation working group" (WG) drives the ExPr forward. The WG consists of key decision-makers from primary stakeholder groups like business units, engineering, and data science, who represent their groups, are accountable for tasks, and may delegate work. Data science plays both a tactical role as statistical experts (senior data science manager) and a strategic program manager role, often filled by a product manager reporting to data science. The WG meets regularly to discuss hypotheses, prioritize experiments, define action items, assign accountability, set timelines, and review results, with detailed design work occurring between meetings. Tactical tips for success include aligning on goals, maintaining a biweekly cadence initially, using agendas, sharing notes, and holding "meetings between meetings" for specific task execution.

Key takeaway

For AI Product Managers or Data Science Managers establishing or optimizing an experimentation program, forming a dedicated, cross-functional working group is crucial. You should explicitly define the group's goals, assign a program manager (ideally from data science), and ensure regular, structured meetings to prioritize experiments and assign clear action items. This approach fosters accountability and accelerates the delivery of business value through data-driven decisions.

Key insights

An experimentation working group, led by a program manager, drives business results through structured, cross-functional collaboration.

Principles

• Cross-functional collaboration is essential.
• Accountability drives task completion.
• Shared mission and vision foster productivity.

Method

Form an experimentation working group with key decision-makers. Meet regularly to prioritize experiments, assign tasks, and track progress. Design experiment details between meetings. Align on goals and maintain a consistent meeting cadence.

In practice

• Assign a data science product manager as program manager.
• Start with biweekly meetings to build cadence.
• Distribute agendas and meeting notes consistently.

Topics

• Experimentation Programs
• A/B Testing
• Data Science Leadership
• Cross-functional Collaboration
• Program Management

Best for: Data Scientist, AI Product Manager, MLOps Engineer

Related on AIssential

• Building An Effective Experimentation Program (Experimentation Program Series: Guide 01) — Building an effective experimentation program requires more than just technical A/B testing knowledge; it demands structured processes and cross-functional collaboration.
• How to Work and Compound with AI — Effective AI collaboration requires structured context, configurable preferences, robust verification, and continuous feedback loops.
• Picking an Experimentation Platform: A Retrospective — Platform selection needs rigorous process, but organizational vision and post-contract change management are critical for true experimentation success.
• ML Experiment Tracking in H2O Driverless AI | Part 10 — Effective experiment tracking is crucial for data science, enabling parallel runs, reproducibility, real-time monitoring, and comparison.
• Inside Google’s System for Coordinated A/B Testing Across Its Global Service Fleet — Google's system standardizes large-scale A/B testing across distributed services, ensuring statistical rigor and minimizing interference.
• Agentic AI in the Enterprise Part 2: Guidance by Persona — Operationalizing agentic AI requires a cross-functional operating model, not just advanced technology.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by ML in Production.