Generative AI Evaluation

As generative AI continues to grow rapidly, evaluating models for quality, safety, and reliability becomes increasingly important. If you are interviewing at an AI company, understanding how evaluations work and what companies consider during the process is essential.

This lesson covers what interviewers expect candidates to discuss, how top tech companies conduct evaluations, the key tradeoffs considered, and common pitfalls in evaluation.

How to show you know evals

You demonstrate strong proficiency to interviewers if you can:

• Show breadth in your understanding of evaluation metrics
• Explain how evaluation datasets are constructed and used
• Clearly distinguish between offline and online evaluations

Evaluation metrics

The field of LLM evaluation is still evolving, so there are a lot of different ways that companies choose to design their evaluations. As a result, it’s important to understand multiple common approaches, as well as their pitfalls.

Scores like BLEU, ROUGE, and METEOR are some early and basic ways of understanding how a simple language model performs.

Because earlier evaluation methods have notable limitations, companies often turn to benchmarks and their own internal metrics to assess LLM performance.

You are not expected to know specific benchmarks, since many are proprietary and vary across organizations, but you should understand the general categories these evaluations fall into.

For example LLaMa breaks it up as so:

In practice, many of the state of the art models (Gemini, Claude, GPT) exhibit similar performance on benchmarks. It’s not usually enough to make a difference in performance for your downstream tasks (caveats: models that are better at specific tasks e.g. coding).

As a result, companies will exhibit domain-specific evaluation metrics that are tied to their business goals.

For example, a healthcare agent that summarizes patient history cares a lot about the factuality of the response. Non-factual responses will hurt user trust and could have downstream negative effects for doctors/patients that are using the agent.

In contrast, a mental health agent might care about the underlying safety and fairness of their response. Similarly, unsafe responses will hurt the user experience and decrease usage.

Evaluation datasets

Evaluation datasets are essential for understanding how a system is tested. They consist of carefully selected inputs along with expected outputs, or ground truths, which are used to measure an AI application’s performance, accuracy, and safety.

The key questions in creating, understanding, and evaluating the evaluation datasets themselves are:

• What is the task?
• What do the inputs and outputs look like?
• What guidelines did the labelers follow when creating the data?

Evaluation datasets usually include examples with a clear expected output. Practitioners then apply different evaluation metrics to compare the model’s response against that expected output. These metrics help determine how well the LLM performs on a specific task, or across many tasks.

Some common benchmarks for LLMs include:

• MMLU (Massive Multitask Language Understanding) - general understanding
• GPQA - multiple choice question benchmark for STEM subjects
• EvalPlus - coding benchmark
• SWE Bench - coding benchmark

In practice, companies often rely on their own datasets to evaluate model performance, because each company’s tasks are specific to its domain. For example, if the model is a healthcare agent, the company might create its own labeled dataset containing the correct answers or facts for each example. This allows them to measure how accurately the model performs on tasks relevant to their business.

Since a lot of these tasks are subjective and difficult, labelers often have different ideas of what the expected output should be.

An important metric that measures agreement across labels is called inter-annotator agreement. The higher it is, the more “certainty” there is across labels.

Offline & online evals

• Offline benchmarks: Used before you deploy a model to production,
• Online metrics: Use to evaluate a model in production.

Companies typically evaluate a model on an offline dataset before deploying it to production. This allows them to determine whether the model is performing well enough to be deployed.

Offline evaluation has its limitations because it only measures performance on data you expect to see in production, often based on historical datasets. However, historical data can differ from real production data.

Evaluating a model online in production provides more accurate insights, but it is more challenging. You may not always have the expected outputs for incoming production data, depending on how your data labeling process is set up.

Common tradeoffs

Human labeling vs. automated labeling

As discussed above, it is important to know the expected output, or “ground truth,” when evaluating model performance. This can be provided by human labeling teams, or by automated systems, such as another LLM that is well-suited for generating labels for your task (also called LLM-as-a-judge).

Offline vs. online

Deciding whether to evaluate a model offline, online, or using both approaches is an important decision that should ideally be made from the very beginning.

Common pitfalls

Offline evaluations not matching in-production behavior

Offline evaluations are useful, but they can only give an estimate of how the model will perform. Once a model interacts with real users, production inputs often look very different: messy formatting, ambiguous queries, missing context, or entirely new edge cases. The problem is not evaluating offline itself, but assuming that offline results will perfectly predict production performance.

Choosing metrics that don’t reflect your actual use case

The easiest trap in Gen AI evaluation is picking a metric because it exists, and not because it’s meaningful for your task. Metrics like BLEU or ROUGE measure similarity, not usefulness. If the metric doesn’t reflect what “good” looks like for your product, it’ll create the false impression of progress because you are optimizing for the wrong behavior.

Low-quality or unrepresentative evaluation datasets

Bad datasets may distort your understanding of the system entirely. If labels are inconsistent, instructions unclear, or examples too narrow, your evaluation pipeline becomes disconnected from reality. High variance in labels, in particular, makes it nearly impossible to know whether the model is wrong or the dataset is.

Slow feedback loops

If evaluation takes days or weeks due to slow labeling, large datasets, or infrastructure bottlenecks, it becomes difficult to iterate effectively. Long delays can result in outdated results, slower debugging, and a higher risk of releasing regressions that go unnoticed for too long.