AI Technical Fluency

This lesson covers the 20% of AI technical concepts that come up in 80% of interviews at many Frontier AI labs and other state-of-the-art companies. It’s a solid primer to start with. Later, if you want to go deeper, check out our Generative AI Interviews course.

Why this matters now

What it means to be technical has changed since the rise of LLMs. To be competitive as a PM today, you need to have AI technical fluency. Whether the top companies say they want it or not, the reality is that they do.

The model lifecycle

Every AI model goes through three phases. Some companies, like Apple, tend to expect their PMs to get into the weeds of AI architecture because product decisions often come down to swapping out components to address different tradeoffs: latency, hallucination, cost, and data privacy. Understanding the lifecycle will help you reason through those decisions, including in interviews.

1. Pre-training is when the model learns language patterns from massive datasets: grammar, facts, reasoning, and relationships between concepts.

   Think of it like a person spending years reading every book, article, and website on the internet before ever starting a job. They haven't been trained to do anything specific yet, but they've absorbed an enormous amount of general knowledge.

   The output is powerful but raw, with no alignment or domain-specific knowledge. Pre-training is extremely costly and is almost always handled by foundation model providers such as OpenAI, Google, Anthropic, or Meta.

   You might also hear this referred to as base model training or foundation model training.

2. Post-training is the broader phase that shapes model behavior after pre-training.

   Think of it like onboarding that new hire: you teach them your company's norms, give them specific tasks to practice, and correct them when they behave in ways you don't want.

   This includes fine-tuning on domain-specific data (legal, medical, enterprise), instruction tuning so the model follows natural language prompts, and RLHF (reinforcement learning from human feedback) to align outputs with human preferences.

3. Inference is when the model responds to a live user input. This is the product your users experience.

   The model doesn't learn anything new here. It's drawing on everything it absorbed during training. Think of it like the employee doing their actual job: every response is a fresh task, but they're working from the same base of knowledge they built earlier.

   Behavior can vary significantly depending on how you prompt it, what context you provide, and the settings you configure.

Tokens & context windows

NVIDIA's senior PM loop regularly asks candidates:

"How do you measure whether an LLM project is efficient? And how do you measure model quality?"

Efficiency is fundamentally about token cost per output and latency per call. Quality is about whether the output is actually right. These aren't separable questions, and understanding why starts with knowing what tokens are.

Tokens are the units a model reads and writes (roughly a word or part of a word in English). Everything that goes into a model (system prompt, conversation history, user input) and everything that comes out costs tokens. This is the basis for how AI APIs price usage and where context limits come from.

Efficiency comes down to two things: how many tokens you're burning per output, and how long each call takes. A rough way to think about it:

• Token efficiency = useful output generated/total tokens consumed
• Latency efficiency = response quality/time to first token (or full response)

In practice, this means asking: are we passing in a 2,000-token system prompt when 400 tokens would do the same job? Are we sending the full conversation history every turn when a summary would work just as well? Waste here compounds fast at scale.

Quality is simply whether the output is actually right for the user. That sounds obvious, but it's a separate variable from efficiency. A model can be cheap and fast and still consistently wrong. You need evals to measure quality independently (covered in the next section).

Context windows are the maximum number of tokens a model can process in a single call; in other words, the model's working memory. Once you hit the limit, older content gets truncated, or you pay more for a model with a larger window.

Hallucinations

When Google's Gemini PM loop asks, "Users are complaining that Gemini is confident but wrong, how would you fix this?", the interviewers are assessing whether you can diagnose the root cause and propose a system-level fix. To do that, you need to understand how hallucinations actually work.

Hallucinations are confident, plausible-sounding outputs that are factually wrong. The model isn't lying; it's pattern-matching from training data without grounding in truth. There are two distinct root causes, and knowing the difference determines your response.

• Training-time hallucinations occur when the model has learned something incorrect, outdated, or underrepresented in the pre-training or fine-tuning data. No amount of clever prompting fixes this. The knowledge isn't there.
• Inference-time hallucinations happen when contradictory or confusing context in the prompt causes the model to fill gaps poorly. This can be improved through better prompt design.

The four design mitigations every PM should have ready:

1. Retrieval-augmented generation (RAG): Instead of relying on what the model learned during training, RAG fetches relevant, up-to-date information at runtime and passes it to the model as context before it generates a response.

   A legal research tool, for example, wouldn't rely on a model's training data to answer questions about case law. It would first retrieve the relevant documents from a database, then generate a response grounded in those documents. If the model gets something wrong, you can trace it back to the retrieved source rather than a mysterious training artifact.

2. Grounding with citations: The model is instructed to make only claims it can back with a source and to surface that source in its response. This makes errors visible rather than hidden inside a confident-sounding paragraph.

   Perplexity does this well: every claim in a response is tied to a URL. Users can verify. And critically, when the model is wrong, the citation makes the failure auditable rather than invisible. This also shifts user behavior: people learn to check sources rather than blindly trust outputs.

3. Confidence thresholds and human-in-the-loop: Not every output needs to go straight to the user. You can build a layer that flags low-confidence responses for human review before they're surfaced.

   A medical symptom checker, for example, might route any response in which the model's confidence falls below a set threshold to a human reviewer or to a fallback message like "we recommend speaking with a doctor." The model still handles the high-confidence, low-stakes cases at scale. The risky ones get a second look.

4. Evals: You can't fix what you don't measure. Evals are a structured way to test your model's outputs before and after any change, specifically looking for hallucination rate alongside other quality signals.

   Before shipping a new model version, you'd run it against a benchmark set of prompts for which you already know the correct answers. If the hallucination rate goes up compared to the previous version, you don't ship. This is how you avoid discovering problems in production through user complaints rather than catching them in testing.

Evals

Evals are the test suite for your AI product. Unlike traditional software, where you can write deterministic unit tests ("if input X, expect output Y"), AI output is probabilistic. The same prompt can return different responses across runs, and "correct" is often a matter of degree rather than a binary pass/fail. Evals fill that gap by giving you a structured, repeatable way to measure output quality.

They typically include three types:

1. Human review: Raters score outputs on quality, accuracy, and tone. This is the most reliable signal, but also the slowest and most expensive. It works best for high-stakes outputs where nuance matters, such as medical assistant work or legal summarization tools. In practice, a team might review 200 sample outputs after every major model update and score each one on a 1-5 rubric across accuracy, tone, and helpfulness.
2. LLM-as-judge: A second model evaluates the output. You prompt it with something like "here is the user's question and the model's response, rate the response on accuracy and helpfulness from 1 to 5, and explain your reasoning." This is faster and scales better than human review, but it introduces its own bias: the judge model has its own blind spots and may systematically favor certain response styles. It works well as a first-pass filter before human review, not as a replacement.
3. Automated metrics: These are scoring algorithms that compare model outputs against a known correct answer without any human or model judgment involved. You define the check yourself based on what "correct" means for your product.

For a coding assistant, this might mean "does the generated code actually run without errors?" For a customer support bot, it might mean "did the response contain the correct policy number?" For a translation tool, it might mean "does the output match the verified reference translation?"

They're faster and cheaper than human review or LLM-as-judge, but only work when there's a clear, verifiable correct answer. They're less useful for open-ended tasks like creative writing or summarization, where quality is harder to define as a binary check.

As a PM, you define what "good" means: factual accuracy, tone match, and latency under a threshold. Evals operationalize those requirements into measurable signals. Any model update, prompt change, or data refresh should run through your eval suite before it ships.

Retrieval-augmented generation (RAG)

RAG is used when you have an existing body of knowledge and want your AI product to search and draw from it accurately. Instead of relying on what the model learned during training, RAG fetches relevant, up-to-date information at runtime and passes it to the model as context before it generates a response. This is especially useful when your data is proprietary, frequently updated, or too specific to be well represented in the model's training data.

A legal research tool, for example, wouldn't rely on a model's training data to answer questions about case law. It would first retrieve the relevant documents from a database, then generate a response grounded in those documents.

RAG works in two stages. Before a user asks anything, you convert your knowledge base into embeddings (numerical vectors that capture semantic meaning) and store them in a vector database.

When a query comes in, it's also converted into an embedding and matched against your stored vectors to find the most semantically similar content. That content gets passed to the model as context, grounding the response in retrieved information rather than training data alone.

The decisions live inside the mechanics:

What to embed and store? Not everything in your knowledge base retrieves well. Long documents with buried information are poor retrieval targets. A product FAQ chunked into individual Q&A pairs will outperform the same content stored as one long document.

How to chunk? Chunking is the process of breaking your source documents into smaller pieces before storing them. You do this because the model can only work with a limited amount of context at once, so you want to retrieve only the most relevant pieces rather than dumping an entire document in. The chunk is too large, and the retrieved context becomes noisy. Chunk too small and you lose the surrounding context that gives sentences meaning. The right strategy depends on content type: a legal contract is chunked differently from a customer support transcript.

What embedding model to use? An embedding model is what converts your text into vectors. Think of it as the translator that turns words into a format the system can search. General-purpose embedding models work well for broad topics, but may underperform on specialized domains where meaning is encoded in technical terminology. For example, a medical knowledge base may retrieve more accurately with an embedding model trained on clinical text than a general-purpose one.

Keeping the index fresh. The index is the stored collection of embeddings that your system searches. If your knowledge base updates frequently, you need a process to re-embed and refresh it. A customer support bot trained on last year's product documentation will confidently answer questions using outdated information. Stale retrieval is a silent failure: the model sounds certain, but it's working from an old context. In practice, this means setting up a pipeline that automatically re-embeds content whenever your source data changes.

When a RAG system returns irrelevant results, the embedding model or chunking strategy is usually the first place to look, not the generative model on top of it.

RAG vs. fine-tuning vs. prompting

Choosing between prompting, RAG, and fine-tuning comes up both when diagnosing why a model isn't performing and when making initial architecture decisions for a new product. It’s reported that Perplexity's senior PM loop tests this directly by asking candidates to explain how RAG works, as it's core to how their product operates.

OpenAI goes even further. In a principal PM interview, candidates were given a take-home asking them to define OpenAI's fine-tuning strategy using only public information.

The judgment required is the same in both cases: match the right lever to the actual problem. Not using the different options available leads to ballooning resources (e.g., cost, time)

Use this decision flow to quickly identify the right approach:

1. Start with prompting. It's fast and often underestimated.
2. Move to RAG if the problem is the model not knowing current or proprietary information.
3. Fine-tune only when you need to have a specialized task that prompting and RAG can't solve, and you have the data and budget.

Agentic AI

Unlike standard AI models that respond to a single prompt, agents take sequences of actions to complete a goal. They can use tools (search, code execution, APIs), maintain state across steps, and adapt based on intermediate results.

A good way to picture this is to imagine asking an AI assistant to plan a business trip. A standard model gives you a list of suggestions. An agent actually books the flights, checks calendar availability, reserves a hotel within budget, and sends the itinerary to your email. Each of those is a separate action, taken in sequence, with the agent adjusting based on what it finds along the way.

At a high level, an agent operates by receiving a goal, breaking it into steps, executing each step using tools, observing the result, adjusting the plan based on what it learns, and continuing until the goal is complete or it hits a stopping condition.

What you need to think about as a PM building with agents:

• Failure modes compound. Each step can fail, and errors cascade. If an agent misreads step two, steps three through ten are built on a bad foundation. Evaluation is harder than for single-turn outputs because you're not just checking one response; you're checking a chain of decisions.
• Human-in-the-loop thresholds. This means deciding up front which actions the agent is allowed to take on its own and which require the user to confirm first. Sending a calendar invite is low stakes. Sending an email to a client, making a purchase, or deleting a file is not. The more irreversible the action, the more you want a human in the loop before it executes.
• Logging agent steps. You need a record of every action the agent took and the reasons for it. Without this, when something goes wrong, you have no way to trace back what happened. It also matters for user trust: if a user asks, "Why did you send that email?", you need to be able to show them the reasoning. Think of it like a flight recorder for your agent.
• Cost control. Agents can spin up long chains of tool calls to complete a single goal. Each call costs tokens. Without limits in place, a single agent run can get expensive fast, especially if the agent gets stuck in a loop or takes an unnecessarily long path to complete a simple task.

MCP (Model Context Protocol)

MCP (Model Context Protocol) is an open standard that lets AI models connect to external tools, data sources, and services in a structured way. Think of it as the API contract between an AI agent and the outside world: it defines how the model requests data from a CRM, runs a search, or triggers an action in another system.

Without MCP, every integration between an AI model and an external tool requires custom code written from scratch. MCP standardizes the contract so the same model can talk to a CRM, a calendar, a database, and a code editor without a bespoke integration for each.

A concrete example: Claude can connect to tools like Google Drive, Notion, Slack, and GitHub through MCP. When you ask it to "summarize the latest comments in my Notion doc and draft a Slack message about it," it's pulling data from two separate systems in a single request. MCP is what makes that possible without a custom integration built between each pair of tools.

For PMs building agentic or multi-system AI products, MCP matters because it standardizes how models access external context (reducing custom integration work), makes tool use more reliable and auditable, and is increasingly expected as a baseline for enterprise AI products.

Try it yourself: Open Claude.ai and go to Settings. Under Connectors, you'll see a list of tools you can connect Claude to, things like Google Drive, Notion, Slack, and GitHub. Connect one and then ask Claude a question that requires it to pull from that source. Notice how Claude retrieves context from an external system and uses it to generate a response. That pipeline is MCP in action.

Temperature & sampling

"How would you make a model's output more creative?" is a question that comes up in product design rounds at companies like Google and Apple. It sounds like a product question, but it's a technical probe. Candidates who answer with "we'd tune the UX" reveal they don't understand how the model itself works. The interviewer is testing whether you know that creativity is a function of temperature.

Temperature is a parameter you set when calling a model. It controls how deterministic or creative the model's outputs are. At low temperature (near 0), the model picks the most likely next token every time: predictable, consistent, reliable. At high temperature, it samples more broadly: more varied, more surprising, and sometimes more wrong.

Here is how that plays out in practice:

Low-temperature example: A user asks a legal document-drafting tool to summarize a contract clause. The model produces the same clean, precise summary every time. There is no room for creative interpretation because accuracy is what matters here. Getting "creative" with a legal clause is a liability.

High temperature example: A user asks a marketing copy tool to generate five tagline options for a new product launch. The model produces five genuinely different options with distinct tones and angles. At low temperatures, the five options would look nearly identical. High temperatures are what make the variety possible.

The practical PM decision is: what does this feature need to be? Consistent and accurate, or varied and generative? A medical assistant who gets creative is dangerous. A brainstorming tool that always plays it safe is useless.

Latency & streaming

Latency is the time between a user submitting input and the model completing its response. For LLMs, this can run into seconds, which is unacceptable for many real-time use cases.

Streaming solves the perception problem by sending tokens to the UI as they're generated. The user sees words appearing progressively rather than waiting for a complete response. Total time to completion is the same, but perceived responsiveness improves dramatically.

You've seen this in action if you've used ChatGPT or Claude: the response appears word by word rather than all at once. That's streaming. Without it, the user stares at a blank screen for several seconds before anything appears, which feels broken even when the model is working fine.

UX implications:

• Streaming is the right default for conversational interfaces
• Batch (non-streaming) fits background tasks like document summarization or report generation
• Long-running agentic tasks need progress indicators, not just a typing indicator

Latency also directly impacts product decisions beyond UX. A real-time medical triage tool that takes 8 seconds to respond is not real-time. A live customer support agent who pauses mid-conversation while waiting for a model call will feel broken to the user, even if the answer is eventually correct. Latency budgets need to be defined upfront, not tuned after launch.

Bias & fairness

Models learn from human-generated data, which means they encode human biases. A model trained on historical hiring data may disadvantage certain demographics. A model trained on English-dominant web text may perform worse in other languages. These are predictable failure modes, not edge cases.

As a PM, your responsibilities break down into four areas:

Identify who gets hurt when the model is wrong. Start by mapping your user population. Ask: Which groups are underrepresented in the training data? Where is the error rate likely to be highest? A speech-to-text product trained mostly on American English accents will perform worse for users with non-native accents. A loan approval model trained on historical data may systematically disadvantage groups that were historically denied credit. The goal here is to name the risk before it ships, not after.

Detect bias through disaggregated evals. Don't just measure average performance. Measure across demographic slices, languages, and accessibility needs separately. A model that is 90% accurate on average can still be 70% accurate for a specific group, and the average masks that. In practice, this means:

• Building a test set that intentionally over-represents edge case groups, not just the majority
• Running your eval suite across subgroups and comparing error rates
• Flagging any group where performance drops meaningfully below the average as a launch blocker, not a backlog item

Prevent reinforcement. Bias compounds over time if you're not careful. If a biased model yields worse results for certain users, those users engage less, which means less data from that group feeds back into your training pipeline, making the model even less representative over time. Breaking that loop means filtering model outputs before using them as a training signal, bringing in diverse human raters for labeling, and auditing your data pipeline for representation gaps on a regular cadence.

Be transparent with users. Tell users when AI is involved in a consequential decision, especially in domains like hiring, lending, healthcare, or content moderation. Users who know AI is involved can push back. Users who have no recourse.

In an interview, a strong answer to a biased question does three things. It names a specific group or scenario that would be at risk given the product context. It describes a concrete detection method, like disaggregated evals or a third-party audit. And it explains how you'd prevent bias from compounding over time through your feedback-loop design. Most candidates stop at detection. The prevention piece is what separates a good answer from a great one.