When answering impact sizing questions as a data scientist, interviewers are looking for more detailed estimates than in BizOps and PM interviews. You’re expected to discuss different data sources and statistical techniques to calculate impact.
Following a structured and logical framework helps showcase your problem-solving skills and analytical thinking.
Below, we’ll guide you through a step-by-step process that you can use to discuss impact sizing in an interview.
Step 1. Define the problem. Ask questions to define the business problem, scope, and KPIs.
Step 2. Identify baselines. Identify an existing comparable or baseline to measure performance.
Step 3. Estimate the equation variables. Estimate the variables from the equation. State your assumptions at each step.
Step 4. Identify and interpret the impact estimate. Estimate the final impact as a range and discuss its implications.
Step 5. Validate results. Identify biases and validate estimates by comparing them to other metrics.
Step 6. Discuss second-order effects. Recommend next steps or future iterations of the analysis.
Step 7. Summarize. Recap your analysis approach, results, and suggestions.
We’ll use this example interview question to walk through the 7-step answer framework: “How much incremental revenue can a food delivery company make if it expands to a new vertical, e.g. grocery delivery?”
Step 1: Define the problem
Understand the business problem and define the scope by asking clarifying questions about the primary goal, expected output, and market size. Identify the key performance indicators (KPIs) or metrics relevant to measuring the impact.
Step 2: Identify baselines
Identify an existing comparable or baseline to measure performance.
Step 3: Estimate the equation variables
Estimate the variables from your equation and state assumptions at each step. Common statistical techniques for estimating variables include regression analysis, A/B testing, and causal inference methods using observational data. Walk your interviewer through what techniques you’ll use and how their specific inputs and outputs can help establish your estimations.
Step 4: Identify and interpret the impact estimate
Quantify the final impact estimate as a range (with lower and upper bounds) rather than a point estimate and avoid using precise numbers (round up or down to the nearest 1000 or million depending on the size of the business). Discuss the implications of the observed impact in terms of practical significance and business value.
Step 5: Validate results
First, address any sources of bias or confounding factors that may affect the interpretation of results. Be transparent about the limitations of the analysis, such as sample size constraints, data quality issues, or external factors.
Then validate your estimate by comparing it to metrics such as the company's market cap. Also consider other ways to validate your analysis, such as running an experiment.
Step 6: Discuss second-order effects
Provide recommendations for the next steps or future iterations of the analysis. Discussing second-order effects demonstrates your ability to think holistically, and provides an opportunity to distinguish yourself from other candidates who might skip this step.
Step 7: Summarize
Clearly communicate your analysis approach, results, and recommendations in a structured and concise manner.
