Setting Metrics

One of the best ways to drive alignment across teams is by setting metrics that accurately reflect the strategic priorities of the company. It’s extremely important to be thoughtful when setting metrics because what you measure, you optimize. Metrics that are easily gameable will not motivate meaningful business growth. A lack of guardrails and counter metrics (which we will discuss in greater detail) will compromise the business’s longer-term health.

BizOps often plays the pivotal role of setting metrics to focus the business, so it’s not a surprise that this topic often comes up in BizOps interviews.

We will talk about three different common frameworks for thinking about metrics that can help with contextualizing and communicating what’s important in different situations.

Frameworks for Metric Setting

There are three common frameworks for thinking about metrics that can help with contextualizing and communicating what’s important in different situations.

Objectives and Key Results

Many tech companies use the Objectives and Key Results (OKR) framework to connect strategic business goals to quantifiable results that an initiative is supposed to achieve. The objective is often a longer-term strategic goal that many teams are working towards. The key result is the metric outcome that different business initiatives are supposed to achieve to contribute to that objective.

For example, for a social media company, an objective might be to strengthen connections between users. The key result would be to increase sharing of content from one user to another by 5%.

Oftentimes, teams might add another layer of initiatives within each key result. So to continue our example, it could bring content that’s been shared more to a more prominent position or improve autocomplete of friend names when selecting who to share the content with.

While BizOps interviews may not explicitly phrase the question this way, being able to distinguish between objectives, key results, and initiatives is very important. Every time an interviewer asks “how would you set success metrics for x initiative,” what they’re really looking to see if whether you have the ability to hypothesize a couple of different key results this initiative might be looking to drive and being able to potentially connect that to the overarching strategic objectives of the company.

To read more about OKRs and how they can be used, check out this article.

Operational Metrics

OKRs provide a great structure for different teams to drive towards the same goal, but that alone isn’t enough when it comes to truly understanding the impact of an initiative on the business. To demonstrate that you really understand the operational complexity, you need to develop a set of operational metrics.

True North Metrics

This is the core outcome that you want a business initiative to drive. What is the business outcome we really want to drive and how does it connect to company-level priorities? The true north is that outcome. For large tech companies, this is at least partially the key result in OKRs and typically comes tops down. For startups, teams have a bit more liberty to define the true north.

A couple of things to consider:

• What is the true impact you want to have?
• How does this true north capture trade-offs? For example, you can increase the number of high-quality users by a little or low-quality users by a lot.
• What is the right decision for the company and how does the true north reflect that?
• What are some ways that this true north can be gamed and how do we prevent that?

However true north metrics alone are not enough for two reasons:

First, while true north metrics are ultimately the outcome you want to drive, they often take some time to move and can lag quite a bit behind business changes. That’s why you need leading indicators, which we call guidepost metrics, to serve as early indicators. Also, true north metrics can be gamed with unwanted behavior. Counter or guardrail metrics are set to monitor for that and guard against it.

Here, let’s consider an example of an email campaign. What’s the purpose of that email campaign? While email sends, opens, and clicks are useful, what really drives business outcomes is likely purchases or subscriptions the users make once they’re on site. Therefore, the true north would be purchases or subscriptions from an email acquisition.

Guidepost Metrics

One common issue with true north metrics is that they can take a long time to realize. Guidepost metrics are early indicators, or leading metrics, of whether the business change you’re making is driving towards the impact that you want. Typically in a user journey or conversion funnel, guideposts are metrics from higher in the funnel. For example, if you want a new email headline to drive more clicks on the call-to-action, you probably want to see more opens first.

The guidepost metrics you set should also reflect the hypothesis you have for why the true north should improve. Let’s continue considering the email campaign example.

If the hypothesis is that we are sending a new email campaign, then we would expect email sends to be a guidepost metric. If we are changing the email headline, then we should see an increase in email open rates, that’s the guide post. If the hypothesis is that a new email body increases call-to-action (CTA) click-through rate (CTR), then email opens should stay the same. The guidepost would be CTR increase for the CTA.

Counter Metrics or Guardrail Metrics

Counter metrics or guardrail metrics are set to prevent unwanted action that can result from gaming the true north. These often measure the quality of the increase in true north. Continuing our example around email campaigns, the counter metrics could be email unsubscribes. If the new email campaign sends a couple more emails, but many of them lead to users unsubscribing to the company’s emails, then the user action is a clear indicator that the email campaign was a negative user experience. So while there might be a few more purchases or subscriptions, you need to weigh that negative user experience signal against the increase in true north.

Forecasting Metrics

Some BizOps roles require forecasting and modeling both business and financial metrics, so depending on the specific of the role you’re interviewing for, you may be asked to do some basic modeling in Excel / Google Sheets.

Take a deep breath - It’s totally fine if you don’t have a ton of modeling experience or know intimately how the company forecasts its business. The most important thing here is to demonstrate your ability to think through the drivers of the business and how your work can play a role to drive desired outcomes.

Recap

Because bizops is often responsible for setting metrics to focus the business, we can almost guarantee that you'll get at least a few questions on metric-setting. In general, it's useful to brainstorm a big list of metrics you may want to track given your prompt, then downselect until you've identified one or two that best measure what you want to track (true north), then identify counter metrics to avoid issues down the road. Articulate the logic behind your choices throughout, and you'll be set.

To demonstrate competency setting metrics:

• Lay out your logic before plugging in any numbers and talk the interviewer through your thinking.
• Make sure you clearly label all your assumptions and share rationale if applicable.
• Talk through any variables or important assumptions that can most materially affect the outcome and how the company can work on improving that variable / assumption.
• Sense check the output and business assessment. Is this good? Bad? How do we know?
• Provide multiple scenarios if you’re not certain about an assumption. Another way to phrase it is “what do we have to believe for us to achieve X outcome?”