How to Answer ML Production Questions
Overview
ML production questions assess your knowledge of moving ML models off your local environment and into a production environment. Relevant topics include model deployment, evaluation on incoming data, continual learning, monitoring, infrastructure, etc. Some sample questions include:
- How do you know when it is time to replace a production ML model?
- How do you know if your model is working well in production?
- What are the differences between modeling in production vs. any other place?
- How do you deploy your model into a production environment?
- How does batch vs. streaming processing work? When do you use one vs. the other?
To prepare for these types of questions, review the terms under the “ML in production” section of our ML Interview Glossary.
How to answer
There’s often no single right answer for ML production questions. Rather than focusing on getting the correct answer, focus on explaining your rationale clearly and demonstrating domain expertise. Ask clarifying questions about the use case, recall prior experiences that are relevant to the given question, and consider how the company’s specific business objectives might affect the context of the question.
Ensure that you can confidently explain any of the monitoring methods that you mention during your interview. Your interviewer may ask you a follow-up question about a type of method you discuss.
Let’s say your interviewer asks,
“How do you know it’s time to replace your ML model?”
You could say,
“Making these decisions depends on the particular situation and use case. You generally want to replace an ML model when its performance in production degrades compared to its expected performance. The expected performance is typically benchmarked by the offline performance metrics on the initial testing data. The metrics that are chosen will depend on the business use case.
In practice, you may not have ground truth labels for production data, making it difficult to calculate the metrics you selected to evaluate the model offline. Without labels, you can opt to monitor input data distributions to gauge whether or not there is a shift in production.
You can also update models based on time intervals (e.g. every 3 months) or choose to update it if you collected more labeled data since your last training (e.g. training data increased by 5%).”
Common pitfalls
- Not understanding the use case that your interviewer is asking about. The decision to update a model is nontrivial and highly dependent on the situation. You should attempt to clarify the use case, whether or not you have labels, and then explain why you think a particular method would be more reasonable than another.
- Mentioning methods without knowing how to apply them. For example, you shouldn’t mention checking for a data distribution shift if you don’t have a rough idea of how you would do this in practice. You should at least be able to discuss which input you would monitor, what statistics you would check, and how you would gauge if there is a significant difference between these statistics.
- Putting too much pressure on yourself to answer “correctly.” There is no single way of knowing whether you should update an ML model in production (at least without more context or significant empirical evidence that it worked in the past). Interviewers who ask this question generally expect you to use your intuition, combined with any domain expertise that you may have.
Senior candidates
As expected, senior candidates have slightly different performance expectations. The more senior the role, the more you’re expected to demonstrate your ability to:
- Build the model infrastructure from end to end
- Gauge the pros and cons of using a particular algorithm
- Integrate your domain knowledge from previous roles
- Describe your experience productionizing ML models in previous roles
- Work cross-functionally with both technical and non-technical stakeholders.