Statistics & Experimentation Interview Questions

Katherine B. - "If we’re using an A/B test we have a few decision criteria that we can use to measure success. If our primary metric has been shown to be statistically significant (and our confidence interval does not cross 0), and the gaurdrail metrics that we created have not been negatively affected, we should consider shipping. If the our p-value is not significant we can still consider shipping beta if the guardrail metrics have not been negatively affected, and we weigh the opportunity cost of not shippin"See full answer

Himani E. - "The cases where data is under heavy outlier influence. Since mean fluctuates due to the presence of an outlier, median might be a better measure"See full answer

Chetak C. - "Probability that one of the coupons is used = 1 - Probability that no coupon is used = 1 - nC0 p^0 * (1-p)^n = 1 -(1-p)^n"See full answer

Lucas G. - "A confidence interval gives you a range of values where you can be reasonably sure the true value of something lies. It helps us understand the uncertainty around an estimate we've measured from a sample of data. Typically, confidence intervals are set at the 95% confidence level. For example, A/B test results show that variant B has a CTR of 10.5% and its confidence intervals are [9.8%, 11.2%], this means that based on our sampled data, we are 95% confident that the true avg CTR for variant B a"See full answer

Lucas G. - "The central limit theorem tells us that as we repeat the sampling process of an statistic (n > 30), the sampling distribution of that statistic approximates the normal distribution regardless of the original population's distribution. This theorem is useful because it allows us to apply inference with tools that assume normality like t-test, ANOVA, calculate p-values hypothesis testing or regression analysis, calculate confidence intervals, etc."See full answer

Emiliano I. - "Look for the main variables and see if there differences in the distributions of the buckets. Run a linear regression where the dependent variable is a binary variable for each bucket excluding one and the dependent variable is the main kpi you want to measure, if one of those coefficients is significant, you made a mistake. "See full answer

Ayushi A. - "total outcomes : 36 total favourable outcomes : 35 probability of favourable outcomes: 35/36"See full answer

Lucas G. - "Null hypothesis (H0): the coin is fair (unbiased), meaning the probability of flipping a head is 0.5 Alternative (H1): the coin is unfair (biased), meaning the probability of flipping a head is not 0.5 To test this hypothesis, I would calculate a p-value which is the probability of observing a result as extreme as, or more extreme than, what I say in my sample, assuming the null hypothesis is true. I could use the probability mass function of a binomial random variable to model the coin toss b"See full answer

Lucas G. - "Type I error (typically denoted by alpha) is the probability of mistakenly rejecting a true null hypothesis (i.e., We conclude that something significant is happening when there's nothing going on). Type II (typically denoted by beta) error is the probability of failing to reject a false null hypothesis (i.e., we conclude that there's nothing going on when there is something significant happening). The difference is that type I error is a false positive and type II error is a false negative. T"See full answer

Mark S. - "E(VAR(X))= VAR(X) VAR(X)= E[(X-E(X))^2] = E[X^2]-E[X]^2"See full answer

Sinchita S. - "Statistical power is defined as the probability that a test will correctly reject a false null hypothesis. In other words, it is the likelihood of detecting an effect (e.g. a real difference between two groups) if one actually exists. It is typically set to 80% meaning that 80% of the time we will can correctly detect a difference between the groups. It is also a critical component of calculating the correct sample size for an experiment. Let's say if we conduct an experiment on a very small sam"See full answer

Zacharias E. - "log(xy) = log(x) + log(y), thus log(120) = log(1 2 * 3 * 4 * 5) = log(1) + log(2) + log(3) + log(4) + log(5). The two are equal."See full answer

Lucas G. - "I'd recommend to adjust p-values because of the increased chance of type I errors when conducting a large number of hypothesis. My recommended adjustment approach would be the Benjamini-Hochberg (BH) over the Bonferroni because BH strikes a balance between controlling for false positive and maintaining statistical power whereas Bonferroni is overly conservative while still controlling for false positives, it leads to a higher chance of missing true effects (high type II error)."See full answer