Design a System to Predict Netflix Watch Times

## Introduction

A system design interview tests your ability to build a system in a practical setting and adapt it to fit the particular business needs of the company and the users. This often requires discussing the tradeoffs between different methods and making concrete decisions. Interviewers generally look for candidates to display holistic knowledge of a system, as well as how decisions in one part of the system can affect decisions in other parts.

In this conversational interview, the candidate explores key considerations of designing a system for predicting watch times. Rather than designing the system end-to-end and following the typical interview framework, the interviewer tests the candidate's understanding of handling large amounts of data, as well as efficient algorithms for scaling predictions for users across different scales (country, region, family, individuals).

## Question 1: Problem scope 

__The interviewer asks, “How would you approach this problem?”__

As with any machine learning system, we’ll need to ask a lot of clarifying questions about the data that’s available to us, as well as the scope of the problem. This includes:

How much data do we have?
What user information do we have access to? Can we get watchtime logs on a per-user basis?
How representative is the data sample that we have of the general population?
How fine-grained does our prediction system need to be?
Do we want to tailor our predictions based on anything more than just good predictions of previous watch times?

Once we have the answers to these questions, we can start thinking about the kinds of systems we might choose to build to predict watch times.

First, we should consider the kinds of data we have available, and what kind of useful features we might be able to pull out from each example. To help organize things, we’ll split features into user features and movie/TV show features. 

User features include things like demographic information, user profile/ID, past interactions, etc. while movie/show (“item”) features include things like genre, actors, length of item, popularity with other users, etc. We might also want to model other time-based features such as time/date of the watch time log and previous activity during the session.

<CalloutBox type="Tip" open={false}>
Interviewers mainly look for the candidate to start from the bottom up and understand the problem before diving into a solution. Clarifying questions are important here, because they help map out what a candidate is thinking about and whether they’re thinking deeply about the problem. Usually, if an interviewer glosses over a particular question (e.g. assuming we have millions of data points from a representative sample), then that means we probably don’t have to worry about those issues. Follow-up questions here might consider things like data privacy, how to deal with collecting data over time, data storage and filtering, granularity of datasets, dataset splits, and more.
</CalloutBox>

## Question 2: Features and tradeoffs

__The interviewer asks, “How would you represent each of these features and what are the tradeoffs between these representations?”__

Most of these features could be represented using one-hot features, i.e. using one feature for each possible actor, genre, series, etc. This allows our model to learn based on the presence or absence of individual features, which may provide strong signals for watch time, and also allows us to add on new features as necessary to our model. One-hot features are easy to implement, since they don’t require learning any kind of continuous embedding space, and small modifications (e.g. using a continuous number from 0-1 to represent how prevalent an actor is within a series) can add additional information. 

However, with too many features, the input to our model can grow dramatically in size, and we might risk overfitting to individual features that appear very few times in the training set. This can become especially bad when we have a huge combination of potential features for each aspect of the problem we want to model. 

One way to fix some of the issues with one-hot encodings is to restrict our options to the top-k actors/genres in order to prevent our features from blowing up. Additionally, learning some sort of continuous embedding space where we put similar actors or films together with a fixed-length representation is great for embedding new actors or features. However, it is also difficult to learn properly. It also ignores the fact that we might want to model items and users jointly.

<CalloutBox type="Tip" open={false}>
This question tests if you’re able to understand not only the high level information about what features are useful, but also how we’d want to get these into an ML system in practice. System design interviews will usually take this approach, discussing the high level abstract concepts, and then drilling down into the specifics of how those concepts interact in a real setting. Make sure that the decisions you make about your system can be backed up with an actual implementation (or at least a reasonable discussion of how someone would go about implementing them).
</CalloutBox>

## Question 3: Model selection

__The interviewer asks, “What kind of model would you use for watch time prediction, considering that we need to accommodate old and new users?”__

The simplest approach here would be to train a single monolith model on all of the features that we have available to us. This allows us to aggregate information across different demographic groups which can give us more learning signal, but may sacrifice performance on smaller subgroups that we care about (i.e. smaller less populous countries). We can opt to train more demographic specific models but risk overfitting if these groups get very small. At the smallest level we want to model a specific user’s interactions but we encounter issues because many users don’t have many data points with which to train a reasonably accurate model with.

<CalloutBox type="Tip" open={false}>
This is a key point in the interview because it’s likely that further questions will depend on the model selection. Pick something that you know well enough to answer questions about, assuming the model is reasonable within the context of the question. Make sure to describe the potential pros and cons of the method you decide on using, as well as how you might work around potential problems. Start from a high level of abstraction and work your way down, taking cues from the interviewer if they’re trying to nudge you in a certain direction.
</CalloutBox>

## Question 4: Unsupervised methods 

__The interviewer asks, “Can you leverage the scale of data to model user behavior in an unsupervised way?"__

An unsupervised approach such as collaborative filtering relies on measuring similarity between users and using that similarity to predict the watch time. For example, if user A is very similar to user B and user B watched this particular movie for a long time, then you might predict that user A will watch for a similar amount of time.

<CalloutBox type="Tip" open={false}>
Although the interviewee started discussing a supervised model, the interviewer seems to be pushing them to more of an unsupervised collaborative filtering based approach. If you don’t know the specifics of collaborative filtering, the interviewer might guide you towards this solution a bit more strongly. This is probably where they wanted to go the whole time, but wanted the interviewee to discuss their thoughts first before prompting. Don’t be discouraged if the interviewer pushes you in a different direction than the one you discussed! These are mostly open-ended questions, so there aren't any wrong answers, as long as you can back up your decisions.
</CalloutBox>

## Question 5: Embeddings

__The interviewer asks, “How would you build embeddings for users and items to measure similarity over?”__

This representation would most likely be a large N users by M items sparse matrix, where we can fill in a value if a user has watched a specific item. The simplest approach would be to set the value to 1 for the [ij] value if user i has watched item j, but obviously this isn’t too helpful for predicting actual watchtime. 

Another option is to instead store the actual watchtime in this value, which allows us to calculate similarity metrics based on watch times and use this to predict for new users. 

Other options could include -1/0/1 for if a user disliked/didn’t rate/liked an item. We can integrate other signals into other matrices as well.

## Question 6: Matrix storage
__The interviewer asks, “How do you anticipate storing this matrix?”__

Storing this matrix is difficult because it’s very large, but we can make our job easier since it’s very sparse and we only need to store a few values per user. We can also shard the data to only perform a similarity search over a small set of users/items that we think our user will be most similar to.

<CalloutBox type="Tip" open={false}>
A more data focused interview might drill into the details of how you would actually implement the sparse matrix, or how to add rows depending on the sharding scheme. Other questions might focus on other kinds of signals we can encode in our matrix and why these might be relevant to the problem at hand, or how to decide how many items to model. A more algorithm focused interviewer might ask more questions about why modeling the user/item space together like this is a reasonable way to solve the issues with the supervised approach.
</CalloutBox>

## Question 7: Similarity metrics

__The interviewer asks, “What kind of similarity metrics would you use to compare users?”__

A few canonical similarity metrics include Euclidean distance, Jaccard distance, cosine similarity, and Pearson correlation. 
Euclidean distance is our standard distance metric but doesn’t match the feature space that we’re considering, since it penalizes differences in watch time and would prioritize users with few items watched. 
1. Jaccard distance is the intersection divided by the union, or a measure of overlap between two users. This works great for binary representations.
2. Cosine similarity measures the angle between two vectors but discards the length information. It works for any vectors and includes direction It is maximally small if vectors point in different directions (one user liked and another disliked)
3. Pearson correlation measures how correlated two numerical values are.

Each metric makes a few assumptions about the kind of data being compared. 
1. Euclidean distance is generally not good for any collaborative filtering problems since we assume very sparse features and matrices. 
2. Jaccard distance requires binary representations and can only compare one-hot features. We can of course turn any continuous representation into a one-hot by thresholding.
3. Cosine similarity works well for continuous embedding spaces where we have captured some notion of positive vs. negative signal (like vs. dislike)
4. Pearson correlation works well when we might expect a linear relationship between vectors (i.e. we want to measure similarity in magnitude as well)

For this particular problem, Pearson correlation makes the most sense since we’re predicting watchtime (a continuous value) and we expect two people with similarly large or small watch times for a show to have similar profiles.

<CalloutBox type="Tip" open={false}>
Discuss the pros and cons of each metric and make an informed decision based on the problem at hand. A more maths/algorithm minded interviewer could follow up on lots of these metrics by asking about metric spaces and why some distances are “good” for certain kinds of representations. They might also ask about learning embedding spaces and how similarity metrics can factor into our loss function or even how we might combine all these approaches together. For a typical interview, mentioning multiple similarity metrics is already a good sign.
</CalloutBox>

## Question 8: Supervised vs. unsupervised

__The interviewer asks, “What tradeoffs does the unsupervised approach make compared to the supervised approach?”__ 

Originally we were motivated to use an unsupervised approach to learn a personalized model for each user without needing a specific supervised model for each user. Collaborative filtering means we avoid overfitting since we’re measuring similarities across all users, and makes it easy to add new features or users. We can also easily add tasks or other signals in order to make other predictions such as what a user might watch next based on their last watched TV show/movie by measuring similarity between items. 

However, we do need to build more infrastructure to perform large-scale similarity comparisons across our users and store the sparse matrices, and we also have a significant cold-start issue where new users do not have any information that can be used to predict watch time. 

A simple way to combine supervised and unsupervised approaches is to use a latent factor model. The latent factor model is a relatively simple linear model that learns a mean term across all users and items, a residual term per item and per user (e.g. how much more above the mean do users typically watch this item/does this user watch in general), as well as a similarity term. The latent factor model also allows us to model time by having our latent factors depend on certain temporal features, and also allows us to take the same approach to add in interactions between users (e.g. social graph information).

<CalloutBox type="Tip" open={false}>
These kinds of questions test if you understand why different scenarios might require different kinds of models, or when these different approaches work well. Your answer will depend on the specifics of the approaches you chose to discuss, but here we discuss how the collaborative filtering approach specifically solved the issues previously discussed to tie back the point that we’re solving a problem we can’t with the supervised approach. Bringing your points back to their original motivation helps to keep you and your interviewer focused on the original problem. 

We only briefly touched on latent factor models here but the interviewer could go into a deep dive on the tradeoffs and design decisions that could be made for these models. For example, we might discuss how specifically to learn and optimize the latent factors, how to add in additional demographic information or additional features, how we might introduce additional factors or change our similarity metric, or how we could combine different collaborative filtering approaches. 
</CalloutBox>

## Question 9: Deployment 

__The interviewer asks, “What issues might you run into during deployment of this system in production and how would you fix them?”__

There are a lot of issues that come up during production, most of which are based on temporal changes.

Different shows change popularity depending on the time of year or even day of the week (e.g. Christmas, Valentine’s day)
Depending on changes to the UI, the way users interact with Netflix could change what items should be prioritized to increase watch time. 
Users preferences shift over time and trends come and go regularly; our model needs to be able to keep up with virality and the fast pace of social media both locally and globally.

Depending on how severe these factors are, we may need to retrain the model on new data or change how we integrate fast-moving trend information. This process relies on collecting high quality evaluation data and regularly testing our model on new data to see whether dips in performance are due to temporary effects (e.g. UI changes, seasonality) that can be integrated into the model, or whether its indicative of a distribution shift that would require retraining or adjusting the model.

<CalloutBox type="Tip" open={false}>
It’s okay to bring up problems that you don’t necessarily have a solution for, but you should aim first to discuss the ones that you can solve. Thinking on your feet will help here, as well as understanding the nuances and different ways you can adapt the method you chose to different scenarios. This is why it's particularly important for these interviews to choose systems and models that you understand well, even if they aren’t necessarily the absolute best for the task at hand, since you’ll be able to discuss the tradeoffs and ways to solve problems that come up more effectively.

An interviewer could go more into depth about database and infrastructure discussions, asking questions about how to store large amounts of data and introduce new users and items, as well as how to set up pipelines to efficiently serve users. For example, what happens when there is a large spike in users during certain times, and how do we load balance across our servers when we  need to calculate similarity metrics efficiently? Are there ways to do approximate metrics that are “good enough?” Other interviewers might aim to discuss specific distribution shifts and how we might try to detect them or fix models on the fly without needing to fully retrain them.
</CalloutBox>

## Conclusion

This interview describes many different systems and the tradeoffs inherent in choosing specific kinds of models. As with any interview, a lot of different aspects of the methods were glossed over or not mentioned in detail. System design interviews are tough, and it can be highly dependent on the interviewer which direction the interview goes. In general, try not to rush your decisions and ensure you justify your thinking with nuanced discussions. 

Design a System to Predict Netflix Watch Times

Community answers

Related questions

Your answer