Evals: How Do You Know Your AI Product is Working?

AI evaluation questions are appearing in PM interviews at the fastest-growing AI companies. This lesson gives you a clear way to answer them, no matter what product you're being asked to evaluate.

These are real questions from real, recent interviews

Before getting into the steps, look at where these questions are actually appearing. These come directly from PM interviews at top AI companies:

• A Principal PM at NVIDIA was asked: "How do you measure whether an LLM or RAG project is efficient, and how do you measure model quality?"
• A Senior PM at Meta was asked: "Design an evaluation framework for ad ranking."
• A PM Intern at Microsoft was asked: "How would you design an AI evaluation system for resumes?"

The surface varies. RAG pipelines, ad ranking, resume screening. But the underlying question is identical in all three:

How would you design an evals suite for this AI product?

What interviewers are scoring

When you get an evals question, interviewers are looking for four signals:

• Domain clarity. Do you understand what this AI product actually does and what "good" looks like in context?
• Metric selection judgment. Reaching for a word-matching metric for a customer support chatbot (where many valid responses exist and exact wording rarely matters) will not work well. For example, “Your order will arrive Thursday” and “Expect delivery by Thursday” are equally correct responses, but a word-matching metric would score them very differently simply because they don't share the same phrasing
• Systems thinking. Can you move beyond a single metric to describe a layered evaluation strategy across development, pre-production, and production?
• Tradeoff awareness. Do you know what your chosen approach costs you? Speed vs. accuracy, automated vs. human labeling, offline vs. online. Senior candidates defend these choices.

How to answer

The four steps below reflect the natural logic of evaluation. You cannot pick a metric without knowing what you're measuring. You cannot build a dataset without knowing what the metric requires. You cannot defend your choices without acknowledging what you're giving up. Skipping steps does not save time. It produces answers that collapse under follow-up.

"You're the PM for a new AI-powered resume screener at a mid-size recruiting company. The tool automatically filters candidates before a human recruiter ever sees them. How would you design an eval system for it?"

We'll use this question throughout all four steps so you can see exactly how a complete answer builds from start to finish.

Step 1: Define what "good" means for this AI product

Before naming any metric, ground yourself in what the AI is being asked to do and what failure looks like. Ask yourself: What is the cost of a wrong output?

For a healthcare summarization agent, a wrong output might mean a doctor misses a diagnosis. For an ad ranking model, it means revenue leakage. For a resume screener, it might mean a qualified candidate gets filtered out, with potential fairness implications. The failure mode shapes the entire eval strategy.

Step 2: Match the metric to the failure mode

This is the highest-leverage step. Interviewers consistently separate candidates who know eval terminology from candidates who understand it, and this is where the gap shows.

Once you know what failure looks like, choose an evaluation approach that would actually catch it. There are three main layers to know:

Automated metrics are scoring algorithms that compare model outputs against a known correct answer without any human or model judgment involved. You define the check yourself based on what 'correct' means for your product. For a coding assistant, this might mean 'does the generated code actually run without errors?' For a customer support bot, it might mean 'did the response contain the correct policy number?' They're faster and cheaper than human review or LLM-as-judge, but only work when there's a clear, verifiable, and correct answer. They're less useful for open-ended tasks like summarization or creative writing, where quality is harder to define as a binary check. Task-specific signals like latency, refusal rate, and retrieval precision also belong in this bucket.

Benchmarks are standardized tests used to compare model versions over time and track regression across releases. For example, HumanEval is a widely used benchmark for measuring coding ability across model versions. They are less useful for domain-specific tasks that don't map to public benchmarks.

Human evaluation is slow and expensive, but it provides the highest signal for anything subjective: tone, helpfulness, safety edge cases, and factual nuance. Inter-annotator agreement (the degree to which different human raters give the same score to the same output) is your calibration signal. Low agreement means your labeling guidelines need work before the scores mean anything. Companies like Scale AI and Surge AI specialize in providing trained human raters for exactly this kind of evaluation work.

LLM-as-a-judge is increasingly common at companies like Snap and Sierra. One LLM grades the outputs of another. For example, you might use Claude or GPT-4 to score whether a customer support response was helpful, accurate, and on-brand, rather than having a human reviewer read every response.

It is faster than human labeling and cheaper at scale, but it introduces its own biases and failure modes: the judge model may be systematically lenient, or favor responses that sound confident even when they're wrong. It should always be paired with a human-labeled calibration set to validate that the judge is actually reliable.

Step 3: Describe how you'd build and run it

Most candidates answer the "what" and skip the "how." This is the step that separates someone who has thought about evals from someone who has actually built one.

A complete eval system has three stages:

Offline evaluation happens before deployment. You build a labeled dataset that represents the distribution of inputs you expect to see in production. Quality matters more than size. A curated set of 500 representative examples with high inter-annotator agreement is more useful than 50,000 noisy labels. Critically: define your acceptance thresholds before you run the eval, not after. If you set the bar after seeing the results, you are not evaluating. You are rationalizing.

For example, if you're building a customer support chatbot, you'd assemble a set of real historical support tickets with verified correct responses, then measure how often the model's answers match those responses. You'd decide up front that the model needs to achieve 90% accuracy on that set before it's considered ready.

Pre-production gates sit between development and shipping. Before any model change goes out, it must pass a regression suite: a fixed set of test cases designed to catch known failure modes. Think of it as the floor, not the ceiling. Passing the regression suite does not mean the model is good. It means it has not gotten worse on the things you have already caught.

For example, if a previous model version had a known failure mode where it gave incorrect refund policies to users, that scenario lives permanently in your regression suite. Every new model version has to get it right before it ships, regardless of how well it performs everywhere else.

Online evaluation begins once you are in production. Shift to live-traffic monitoring. Depending on the task, this might be user feedback signals like thumbs up/down or correction rate, automated classifiers running on sampled outputs, or periodic human review of flagged interactions. The goal is not to evaluate everything. It is to catch drift and new failure modes before they become incidents.

For example, if you're running a legal document summarization tool, you might sample 2% of live outputs each week for human review, while also tracking thumbs-down rates in real time. If either signal spikes, you have an early warning before users start churning.

Step 4: Name the tradeoffs and defend your choices

This is the seniority signal. Senior candidates proactively call out the limitations of their choices and why they're making that bet anyway.

Every choice trades something. Automated vs. human: speed versus signal fidelity. Offline vs. online: iteration speed versus real-world truth. High precision vs. high recall: the right answer depends entirely on what a wrong output actually costs in your specific product.

Pick the approach that fits the product context you established in Step 1 and say why.

Common pitfalls

Picking metrics before understanding the product. Saying "I would measure F1 score" before explaining what the AI does is a red flag. Metrics only make sense relative to what you are measuring. Start with the product. The metric follows.

Treating offline eval as the finish line. A model that performs well on your test set and fails in production is not a model that works. Always describe a path from offline to online evaluation. Google DeepMind interviewers specifically probe whether candidates understand how offline and online results can diverge.

Listing every metric you know. Breadth signals that you have read about evals. Depth signals that you have done them. Pick two or three approaches, explain why they fit, and defend the tradeoffs.

Skipping dataset quality. Evals are only as good as the labeled dataset behind them. If you do not mention dataset construction, coverage, quality, and inter-annotator agreement, you are describing an eval without its foundation.

Not setting thresholds in advance. Deciding whether a model passes after you see the results is not evaluation. It is a post-hoc justification. Mention that thresholds are set before the eval runs. It is a small detail that signals genuine experience.