Hypothesis Testing and p-values

Hypothesis testing is the process of assessing whether the data supports a specific claim or hypothesis. Typically, there are two groups involved: a control group and a treatment group. For example, in testing whether a new feature increases daily active users (DAU), one group uses the current application and another the new version. We then compare the results.

We define:

• Null hypothesis: Assumes no significant difference between groups.
• Alternative hypothesis: Assumes there is a significant difference.

The p-value is the probability of observing the given data if the null hypothesis were true. A small p-value (typically < 0.05) suggests the observed effect is unlikely under the null, prompting rejection of the null in favor of the alternative. If the p-value is large, we fail to reject the null and conclude there is insufficient evidence to say the treatment has an effect.