Introduction to Experimentation

What you'll learn in this module

The most reliable way to gather data on a potential change to your product is to run a controlled experiment. With software products (as well as ads, websites, and marketing emails), A/B testing gives you this power. If you're the first PM on your product (or even at your company) to A/B test, getting set up and up to speed can be arduous, but it's well worth the effort. In this module, we help you understand how to use the power of A/B testing — and how to get started. We'll cover:

• When should you run A/B tests? A/B tests are good for answering some kinds of questions, and not so good for answering others.
• Equipping your product for A/B testing. Tests don't yield good data unless done right. With your company's in-house platform or an off-the-shelf solution, you can segment users, test out different versions of the product, and view the results.
• Planning and running A/B tests. Once you're set up on an A/B testing platform, what steps should you follow for each test you run?
• Evaluating the results of A/B tests. A/B testing discussions will stretch your memory of that one undergrad statistics class. We give you a quick refresher and help you figure out what your A/B test results mean. (Should you launch the tested feature or not?)
• Pitfalls of A/B testing. A/B testing isn't a panacea, and it's easy to mess up the design, execution, or interpretation of your tests. We share common mistakes to avoid.

We won't teach you the nuances of specific A/B testing platforms, as those vary company to company. We also won't teach you all the statistics needed to plan A/B tests with no help. (Those are complicated, and your platform should handle the calculations for you.) But when you finish the module, you'll have a road map for planning, running, and understanding your first experiment.

Your A/B testing colleagues

• If your company has a data scientist (or data science team), they'll be your best friends during the A/B testing process. They can inform you about company standards for statistical power and significance, do any statistical calculations beyond what your platform provides, and regale you with tales of others' A/B testing failures.
• If you're at a larger company with a homegrown A/B testing platform, the infrastructure team that built and maintains that are also good people to know, as they may understand nuances of the platform's workings that influence your tests. If you're buying an off-the-shelf platform, the vendor's customer success representative (or account manager) can help you learn the platform's advanced features and unlock more value.
• Your engineering team and UI/UX design team will need to build out any variants you experiment with, so they'll be involved in the testing process (and you should plan on A/B tests taking up some of their working time). They can also be a great source of ideas for variants to test out!
• You may wish to involve your user researcher in A/B testing, to concurrently gain small-sample qualitative impressions of a variant you're A/B testing.

What to expect when running A/B tests

Running A/B tests is an exercise in patience and prioritization. You'll want to test many things; you can test only a few at a time. You'll want answers quickly; most tests will take weeks to generate significant results. You should also be prepared for ambiguous results. Some changes will boost the target metric but tank a core product KPI. Many tests will have a positive but statistically insignificant impact that likely just reflects a novelty effect.

It's not all frustration, though: the excitement of finding a change that unambiguously knocks it out of the park is tremendous. And with A/B tested product improvements, it's easy to pinpoint the impact of your choice on product outcomes (great for both your ego and your career).

Why A/B testing matters

Despite their (severe) limitations in the types of changes they work well for, A/B tests are unparalleled in their ability to provide clean data. They allow you to try out (and launch) small but real product improvements. Optimizing many parts of your product in small ways can lead to a large aggregate impact on key metrics like DAUs, conversion rates, and revenues. The sorts of questions you can answer with A/B tests include:

• If I change the placement of this button, will more people make a purchase?
• Which marketing email headline gets the highest open rate?
• Would changing the page image cause people to stay on the page longer?
• Which button text and color combination garners the most clicks?
• Which feature callout boosts feature discovery most?