How to Answer A/B Testing Questions

A/B testing questions test more than your knowledge of experiment mechanics. They test whether you can make decisions under uncertainty, navigate results that don't give you a clean answer, and hold a defensible position when the interviewer pushes back.

How this shows up in real interviews

• Uber: Embedded in a product design question. After designing a gift card redemption flow: "How would you design an A/B test for this?"
• OpenAI: Framed as a rollout question. Lead the launch of GPT-6 into ChatGPT, requiring you to reason through phased rollout, metric selection, and launch decisions under uncertainty.
• Capital One: Direct. "How do you A/B test a new feature?"

The pattern: none of them test the mechanics of experimentation in isolation. Even the plainest version is really asking how the experimentation drives a decision.

What interviewers are looking for

• Investigation skills: Do you scope the experiment before jumping to setup, checking your assumptions about the product, the population, and what's actually being measured?
• Metric judgment: Can you choose a primary metric that maps to your hypothesis, reject the obvious-but-wrong alternative (usually ratings), and name where your chosen metric could mislead you?
• Execution: When results are mixed, can you commit to a ship decision with explicit reasoning, a rollout plan, and a specific reversal condition?
• Communication & collaboration: Do you make your reasoning audible at each step, so the interviewer can follow the logic from hypothesis to impact without having to ask?
• Curiosity & passion: Do you push past the mechanics to ask the harder questions about what the test can't see, where the offline/online gap shows up, and what the experiment would miss about AI output quality?

The 5-part framework

These five components map to the five things an interviewer needs to believe before trusting your recommendation: that you're testing the right thing (hypothesis), in the right way (methodology), measuring the right signals (metrics), drawing the right conclusion (impact), and being honest about what you might be wrong about (tradeoff).

1. Hypothesis

Before touching any setup, state your hypothesis. What do you believe is true, and what result would prove or disprove it?

A strong hypothesis has two parts. The prediction is what you expect to happen. The rationale is why, specifically what user behavior or friction makes you confident this change will matter. Most candidates only state the prediction. The rationale is what separates a hypothesis from a guess.

The rationale names a specific observed behavior, not just a general logic. That specificity is what makes it defensible when the interviewer pushes back.

2. Methodology

Define the experiment setup clearly across four components.

Treatment and control. Be specific enough that an engineer could build the treatment without needing to ask clarifying questions. Always define what the control group experiences. Without it, you can't attribute any change to your intervention.

Population. "All users" is almost never right. Name the segment most likely to be affected, explain what you're excluding and why, and state your rollout percentage with logic behind it.

Duration and sample size. Two numbers to state explicitly: how long you'll run the test, and what percentage of users you'll include. Then justify both.

The floor is usually a minimum of 2–3 weeks, long enough to survive the novelty effect and capture full weekly cycles. For sample size, name your confidence level (typically 95%) and the minimum lift you care about detecting (usually 3–5%).

The thing most candidates skip is explaining why they chose their numbers. "I'd run for three weeks at 5% of users" is weaker than "three weeks gets us past the novelty effect, and 5% of this product's MAU is more than enough volume to detect a 3% lift at 95% confidence."

3. Metrics

What metrics will actually convey useful insights to your product and engineering teams? You need to name the signals that tell you whether the experiment worked and anticipate the metrics that might reveal unintended side effects.

Name three tiers:

Metrics for AI products

For AI features, the standard three-tier framework isn't enough on its own. The core problem is that classic engagement metrics can't distinguish between a user engaging with the correct answer and one with the wrong one. They look identical in behavioral data.

This means you need to add a parallel layer of measurement that tracks whether the AI is actually doing its job well, not just whether users are clicking:

• Output quality: hallucination rate, task completion rate, and factual accuracy. These almost always require offline evaluation, meaning sampled human review of outputs, running alongside the experiment.
• Trust: user override rate (how often users edit or dismiss the model's output), escalation to a manual path, and return rate to the same query.
• Latency and cost: p50/p95 response time, cost per query. AI features often trade speed for quality, and that tradeoff has to be measured explicitly, or you won't see it.

This is what's called the offline/online gap. Your A/B test tells you how users behaved. It doesn't tell you whether the model's outputs were accurate or safe. If your entire metric stack is behavioral, you're testing half the thing. Running a parallel offline eval on sampled outputs is what closes that gap.

4. Impact

Running an experiment just tells you information. The impact section is where you explain how that information will actually drive a decision.

Name the specific metric movement that would make you launch, and the specific movement that would make you not. Then tie both back to the company's broader goal, not just the experiment's goal.

Three outcomes to address explicitly:

5. Tradeoffs

Every experiment has something it can't see. The data tells you whether a metric moved. It doesn't tell you whether the change was actually good for the product.

Name at least one thing your experiment would miss before the interviewer asks. Strong candidates think beyond the metric stack to the qualitative and second-order effects that don't show up in a dashboard.

Common things data doesn't capture:

• Meaningful interactions vs. hollow ones. A feature might increase engagement volume while degrading the quality of that engagement. More comments don't mean better conversations. More clicks don't mean more delight.
• User delight and trust. Satisfaction, confidence in the product, and emotional response to an experience are real outcomes that influence retention, but they don't map cleanly to behavioral metrics. A change that appears neutral in the data might quietly erode how users feel about the product.
• Cannibalization of other surfaces. Making one element more prominent often comes at the expense of something else. A larger button increases CTR on that button, but what did users stop clicking instead?
• Long-term behavior change vs. short-term novelty. An experiment window captures a snapshot. It can't tell you whether the change builds a lasting habit or just produces a spike that decays.
• What you can't A/B test at all. Some changes are too large, too infrastructural, or too brand-defining to test in a controlled experiment. For those, data informs the decision but can't make it.

The move interviewers are looking for: name the specific blind spot that's most relevant to your experiment, not a generic list. What would a purely data-driven analysis of this particular test miss?

Common pitfalls

Jumping to set up before stating the hypothesis. State what you believe is true and why before describing any experiment design.

Picking ratings as your primary metric. For AI features, a behavioral metric like conversation continuation rate or task completion rate is a stronger signal.

Treating "all users" as a valid population. Name the segment, explain why it's right, and give the rollout percentage with logic.

Hedging on the ship decision. Name the framework for resolving the tension, name what you're consciously deprioritizing, and commit.

Missing the offline/online eval gap. If your metrics are entirely behavioral, you're testing half the thing.