A/B Test Google's Homepage

Approach

A/B test questions fundamentally test your ability to use experimental data to make product decisions. For each proposed experiment, you should list:

• Hypothesis: What’s your initial guess on an experimental outcome? E.g. Would increasing the button size drive higher click through rates (CTR)?
• Methodology: How will you test this hypothesis? E.g. increase the button size for 50% of users for a 50/50 A/B test.
• Metrics: What metrics are you concerned with? E.g. CTR on the button.
• Impact: How will experimental outcomes affect the product? E.g. we’ll decide to make the button bigger if the CTR rate is higher with the bigger button than the smaller button.
• Tradeoff: Every A/B Experiment has some sort of tradeoff. What are some potential pitfalls to launching your proposed experiment that might not be evident via a purely data-based analysis? E.g. perhaps making the button bigger drives higher CTR on the button in question, but decreases CTR on other important buttons on the page.

The optimal structure for an A/B PM question can be found in our A/B question guide. Take a look if you need a refresher.

Brainstorm

Let’s start by brainstorming which elements are available to us that we can alter to increase search traffic.

• Google logo
• Search box
• Search button
• Footer
• Profile icon
• Background

This exercise helps us brainstorm what toggles we have to play with, and might spark some ideas.

The Experiments

There are many approaches to this particular question. Below are 3 different experiments, ordered from simple to complex, but there are much more worth considering. Complex answers aren’t always better here - the best answers are sophisticated, clearly explained, and high impact.

1. Make the search box more noticeable

Hypothesis The search box may not be noticeable to users, so we should see if making it more noticeable drives increased engagement. Currently, the mobile google.com home screen is relatively crowded, with several buttons and icons all over the page.

Methodology We can accentuate the presence of the search box with a few approaches. Let’s focus on making the borders of the search box bolder, slightly coloring the search box, and making the search button thicker and larger. Note that in this interview question, our segment is all users. In other A/B testing questions, proper user segmentation may be necessary.

Metrics:

• CTR on search box
• Searches initiated from the search box given an impression
• CTR on other elements of the page
• Bounce rate

Impact: Ideally, we’d see an increase in our first two metrics - more clicks on the search box, and more searches initiated from the search box. We’ll likely expect to see a decline in other elements on the page, but we want to make sure our bounce rate doesn’t substantially increase - this is a sign that our experiment actually just causes users to be more confused at the screen and leave the Google experience entirely.

Tradeoff: Even if our experiment is successful and we launch it, it’s important to check for qualitative tradeoffs with this feature. For example, we may be discouraging users from using image search. This may not show up as a huge negative loss in our metrics, but it’s possible we’re funneling users into the search experience and preventing them from noticing other modes of searching. Another tradeoff might be that we're discouraging users from noticing the Google Doodle. The Google Doodle is important branding material for Google’s appeal as a creative, educational company. We should observe how this feature may diminish engagement with the Doodle.

2. Richer zero state queries

Hypothesis Users get to the google.com homepage, but then don’t know how to use Google to get the answer they want. For instance, maybe a user doesn’t know how to formulate their query.

Methodology Let’s add a feature that addresses this, by providing “richer” zero-state autosuggest queries when users click on the searchbox. (Zero state refers to the state the search box is in when nothing is typed in it.) The idea here is that users click the search box, then abandon their search since they aren’t sure what exactly to type. What if, when users click the searchbox, we show helpful search options. Let’s start with something easy: trending search queries at the time (e.g. “Donald Trump Election”). Let’s add a new auto-suggest query that shows a trending search result in the top position of the search box, such as in the image below:

Metrics:

• % Abandon search query after clicking the search box
• CTR on the new suggested “trending query”
• CTR on the other suggested queries

Impact: Ideally, we would notice an increase in clickthrough rate on the new suggested query. We wouldn’t want to launch this experiment if we’re noticing the CTR on the suggested trending query is significantly lower than the other suggested queries, or if our overall clicks on suggested queries declines. Our principle here is that we want the newly introduced query to produce a net increase in search metrics.

Tradeoff: One potential pitfall with launching this feature is the possibility of inappropriate trending query suggestions. Perhaps very violent news is trending, and users may be shocked to see triggering results in their Google search homepage. To mitigate this issue, we should ensure that our safety and abuse algorithms can reliably detect inappropriate content.

3. Include shortcut links on the homepage

Note: this proposed experiment is an actual newly launched feature for the Google mobile homepage.

Hypothesis When users visit google.com, they aren’t sure exactly how to search or how to find some of Google’s most useful features. For example, users may never even know that you can search for the weather in Google. What if Google's homepage surfaced useful content categories to users? This might help users discover new features and begin their search queries.

Methodology Let’s design a series of icons that appear below the search box to show useful features contextually. For instance, we can add a “weather” icon and other useful icons on the homepage. We can update these icons from a set of hundreds of different useful queries (e.g. if the user clicks entertainment, we can surface more entertainment-related queries). An example is shown below:

Metrics:

• CTR on the various icons
• Search engagement (e.g. abandon rate, number of follow-up queries) given a user clicks the new icons
• CTR on the searchbox

Impact: If our searchbox interaction rate declines by a rate greater than the gain from the icons' CTR, then we may not want to launch this feature. Our goal is to increase the user interaction rate on the mobile homepage. Ideally, we’d see a slight decline in the searchbox usage, with a large increase in the new icon CTR, for a net gain in interaction rate.

Tradeoff: One potential flaw: users may get comfortable clicking icons instead of typing queries. While normally users search for a variety of queries ("weather tomorrow" "weather in Maine", etc.) users will only search "weather" via the icon click. This could have complex and deleterious effects on long term Google search retention.

Overall, any of these experiments would be great to discuss with a high level of detail and foresight.