Interview Questions

Katherine B. - "If we’re using an A/B test we have a few decision criteria that we can use to measure success. If our primary metric has been shown to be statistically significant (and our confidence interval does not cross 0), and the gaurdrail metrics that we created have not been negatively affected, we should consider shipping. If the our p-value is not significant we can still consider shipping beta if the guardrail metrics have not been negatively affected, and we weigh the opportunity cost of not shippin"See full answer

Katherine B. - "We want to use rigorous framework for evaluating shipping a new feature — ideally an A/B test. If an A/B test is not available, we first evaluate quantitative data; we look at feature adoption metrics, time-to-use, retention and frequency of visitation. What does the business impact of the feature on conversion rates, revenue per users and LTV, and secondarily evaluate any error rates that could be occurring after the launch of the new feature. It’s important for this analysis to perform segmen"See full answer