AI engineer interview questions show up inside software engineer, ML engineer, and forward-deployed loops at AI-first companies.

They test whether you can build and serve an LLM-backed product under real constraints: inference batching, RAG, agent design, cost and token budgets, and evaluation.

Every question below is pulled from real interviews candidates reported to us at OpenAI, Anthropic, Scale AI, Sierra, xAI, Databricks, Perplexity, and others. Browse them all in our question bank.

✅

Verified: Sourced from real candidate-reported interviews. Read more real interview experiences from AI engineers.

LLM and GenAI concepts

Conceptual questions screen whether you understand what's under the API call.

The clearest real example is Sierra asking candidates to explain retrieval from scratch.

Explain how RAG works. (Sierra)
What GenAI skills do you think will be critical in the next year? (Anthropic)
What is a tokenizer, and why does token count drive both cost and context limits?
What is an embedding, and what does semantic similarity mean in a retrieval system?
What is a context window, and what breaks when you exceed it?
What does the temperature parameter control, and when would you set it near zero?
What is quantization, and what do you trade away when you quantize a model?
Prompting vs. RAG vs. fine-tuning: how do you choose?

Model answer: "Explain how RAG works."

Retrieval-augmented generation gives a model facts it wasn't trained on by fetching them at query time. Four stages: ingest source documents by chunking them and embedding each chunk into a vector store; retrieve by embedding the user's query and running a similarity search for the top-k chunks; augment the prompt by inserting those chunks with instructions to answer only from the provided context; generate the answer, ideally with citations. The reason to use it over fine-tuning is freshness, you update the knowledge by re-indexing instead of retraining. The part interviewers listen for is that retrieval and generation fail independently, so you evaluate and debug them separately.

Sierra asked this as an opener in a software engineering loop, which is the pattern across AI-first companie.

The "AI engineer" screen is a SWE screen with retrieval and serving questions layered in.

ℹ️

Read more: Community answers for how candidates structured their responses to this question.

LLM concepts interview tips

Define the concept in one sentence, then give a concrete example. "An embedding puts 'cancel my plan' and 'how do I unsubscribe' close together in vector space" beats a paragraph of theory.
Tie each concept to a trade-off, since that's what separates someone who's read about RAG from someone who's shipped it.
For anything about model output, note that low temperature isn't the same as correct. A confidently wrong answer at temperature 0 is still wrong.

ℹ️

Practice GenAI concepts: GenAI interviews course →

LLM serving and inference infrastructure

This is the most distinctive AI-engineering category, and the questions are remarkably consistent. Expect to batch requests to maximize GPU utilization without breaking latency.

Anthropic asks versions of it in SWE, ML, hardware, and EM loops.

Design an inference batching system for a single GPU that can handle up to 100 inputs per batch while users wait synchronously, maximizing utilization under compute constraints. (Anthropic)
Design an end-to-end batching system for LLM queries. (Anthropic)
Design an API that lets users sample from large language models efficiently, with good batching and request orchestration. (Anthropic)
Design a system that supports ML models. (Scale AI)
Implement a GPU credit management system. (OpenAI)
Design the OpenAI Playground. (OpenAI)
Design a document processing pipeline that ingests and indexes large volumes of heterogeneous documents for LLM use (Databricks).
Design an ML experiment tracking and analysis platform (Google).

Model answer: "Design an inference batching system for a single GPU (up to 100 inputs/batch, synchronous users)."

The tension is throughput versus latency: bigger batches use the GPU better but make early-arriving requests wait. I'd use dynamic batching with two triggers, a max batch size (100) and a max wait window (say 5 to 20ms), firing the batch when either hits. Incoming requests land in a queue; a scheduler pulls up to 100, pads them to a common shape, runs one forward pass, then scatters results back to the waiting callers. Knobs I'd call out: the wait window as the latency-throughput dial, handling variable sequence lengths (bucket by length so padding doesn't dominate), and backpressure when the queue grows so latency stays bounded. For LLM token generation specifically I'd mention continuous batching, where finished sequences leave the batch and new ones join mid-generation, since fixed batches waste compute when sequences finish at different times.

Interviewers want the latency-throughput trade-off made explicit and at least one production detail (continuous batching, padding by length, backpressure).

Infrastructure interview tips

Lead with the latency-throughput-cost triangle and name your dials: batch size, wait window, model size, caching.
Mention continuous batching and KV-cache reuse for token generation. They signal you've thought about LLM serving specifically, not generic request batching.
Design backpressure and the overflow path, not just the happy path.

ℹ️

Practice with real questions: Browse OpenAI questions →

RAG and agentic system design

Agent design is now a standard prompt, especially at companies whose product is an agent (Sierra, Scale, Glean).

The questions reward guardrails, cost control, and a human fallback.

Design an insurance-claims agent that ingests claims and outputs an approval decision using RAG and infra/storage choices, while controlling LLM/token cost. (Scale AI)
Build your own customer service AI agent for a hypothetical outdoors company. (Sierra)
Design an agentic AI system to power customer support for Spotify. (Sierra)
Design an AI agent for a streaming service. (Sierra)
Design an AI system that integrates with a third-party company's data and workflows. (Sierra)
Have you built any end-to-end agentic systems? (Glean)
Design a file uploader for an AI chat app that handles large multimodal uploads and feeds them into a model's context (xAI).
Design a real-time voice AI application.

Model answer: "Design an insurance-claims agent using RAG while controlling token cost."

Scope first: input is a claim plus supporting docs, output is approve / deny / escalate with a reason. Pipeline: ingest and index policy documents and claim history into a vector store; for each claim, retrieve the relevant policy clauses; pass the claim plus retrieved clauses to the model with a structured-output instruction. Three things this question is really testing. Guardrails: the model proposes a decision, but anything above a confidence or dollar threshold routes to a human, because a wrong auto-approval is expensive. Cost control: cache embeddings, retrieve a tight top-k instead of stuffing the whole policy, route easy claims to a smaller/cheaper model and reserve the large model for ambiguous ones, and cap tokens per call. Evaluation: measure retrieval (did we surface the right clause?) and decision accuracy separately against an audited set, and log every decision with its citations for appeal. I'd state the token budget per claim as an explicit SLA.

The strong move is treating cost and guardrails as first-class design constraints rather than afterthoughts. See the community answer.

Agentic design interview tips

Always design the human fallback and the guardrails (out-of-scope refusal, output validation, confidence thresholds). Agent questions are really risk-management questions.
Make token and latency cost explicit: caching, tight top-k retrieval, model routing, per-request budgets.
Separate retrieval evaluation from decision/answer evaluation, and log decisions with citations for auditability.

ℹ️

Practice agentic design: Browse Sierra questions →

Model behavior, data, and evaluation

ML-engineer loops at AI labs probe whether you can diagnose a misbehaving model and wrangle messy data, not just call an API.

A deployed conversational model gives confident but factually wrong answers in high-risk contexts. How would you investigate and mitigate it? (Anthropic, ML engineer)
Extract and clean a usable dataset from a company demo database using only SQL and Python. (Anthropic, ML engineer)
How would you evaluate an LLM feature when there's no single correct answer?
What does LLM-as-a-judge do, and what are its failure modes?
How do you catch regressions when you change a prompt or swap a model?

Model answer: "A model gives confident but wrong answers in high-risk contexts. Investigate and mitigate."

First I'd quantify it: build an eval set of high-risk prompts with known answers and measure how often the model is both wrong and confident, because "usually" needs a number before I can tell if a fix helped. Investigation: is it a retrieval gap (the model never had the facts), a calibration problem (it's wrong and confident regardless), or a prompt problem (we're not telling it to express uncertainty or abstain)? Mitigations, layered: ground high-risk answers in retrieval with citations so claims are checkable; instruct and few-shot the model to say "I'm not sure" and to refuse outside its competence; add a verification pass or LLM-as-judge that checks the answer against sources before it ships; and for the highest-risk contexts, route to a human. Then I re-run the eval to confirm the confident-wrong rate dropped without nuking helpfulness. The framing that matters is that confidence and correctness are separate axes, and the dangerous quadrant is confident-and-wrong.

This is the question that most rewards production experience. Anthropic asks it in ML loops.

Evaluation interview tips

Quantify before you fix. An eval set turns "it feels wrong" into a number you can move.
Separate the failure modes: retrieval gap vs. calibration vs. prompt. Naming them shows a debugging method.
Validate any LLM-as-judge against human labels, since judges drift and carry biases.

ℹ️

Practice ML and evaluation questions: Browse Anthropic questions →

AI-assisted and practical coding

Coding rounds at these companies are project-style: build a working feature, debug unfamiliar code, often with an AI assistant allowed in the IDE. These are all real, recent questions.

Model answer: "How should you use an AI assistant during a coding round when it's allowed?"

Treat it like a fast pair, not an oracle. State the plan out loud first so the interviewer sees my reasoning, then use the assistant for scaffolding and APIs I'm fuzzy on, and read every line before running it. The failure mode that loses offers is pasting code you can't explain, so when the assistant is wrong I say so and fix it, which is itself signal. I lean on it for the tedious parts (parsing instruction files, test scaffolding, edge cases) and keep the core logic, the part actually being evaluated, in my own head. Anthropic's image-pipeline and Sierra's retry questions are graded on a working feature that holds up under edge cases, so I leave time to handle empty inputs, malformed files, and large sets, and to write at least one test.

Practical coding interview tips

Narrate the plan before you write or prompt. An AI assistant can hide your reasoning if you let it.
Read everything the assistant generates. Explaining and debugging its output is the real test.
Handle the boring cases on purpose: empty input, retries, malformed rows, large files. Project questions reward robustness over cleverness.

ℹ️

Practice project-style coding: Take the practical coding interviews course →

GenAI strategy and behavioral

Even in engineering loops, AI companies probe judgment about shipping AI responsibly and staying current. Anthropic's behavioral rounds in particular lean hard on values and safety.

Develop a strategy for OpenAI's fine-tuning capabilities using publicly available docs. (OpenAI)
How would you define success metrics for an AI-oriented feature or product? (Perplexity)
How do you approach GenAI safety in consumer products? (OpenAI, Anthropic, Google)
Tell me about a time you were confident in a solution and later realized it was wrong. (Anthropic)
Tell me about a time you had to build something that conflicted with your personal values. (Anthropic)
Why do you want to work at Anthropic? (Anthropic)
What are some new advancements in AI you find interesting? (Anthropic)
How would you explain a technical concept to a non-technical person? (Anthropic)

Behavioral interview tips

Use STAR and put a number on the result. "Cut p95 latency from 4s to 900ms" beats "it got faster."
At safety-driven companies like Anthropic, expect values questions in technical loops. Have honest, specific examples ready, not rehearsed mission-speak.
For "confident and later wrong," show the debugging path and the lesson, not a story where nothing went wrong.

ℹ️

Practice behavioral interviews: Start a mock interview →

AI engineer interview frameworks

RAG / agent design framework

Use this to structure any retrieval or agent question so you don't skip a stage.

Stage	What to do
Scope	Inputs, outputs, latency budget, cost ceiling, what a wrong answer costs
Ingest	Chunk and embed source docs into a vector store with metadata
Retrieve	Embed the query, pull a tight top-k (hybrid search if needed)
Act / generate	Structured output; for agents, define tools and approval gates
Guardrails	Refuse out-of-scope, validate output, human fallback above thresholds
Evaluate	Measure retrieval and decision/answer quality separately; log with citations

LLM serving framework

A repeatable order for inference and serving questions.

Step	What to cover
Requirements	QPS, latency SLA, sync vs. async, batch size limits
Batching	Dynamic batching (size + wait window), continuous batching for generation
Efficiency	KV-cache reuse, padding by sequence length, model routing
Backpressure	Queue limits, overflow behavior, bounded tail latency
Cost	Token budgets, caching, smaller models for easy traffic

STAR (behavioral)

Step	What to do
Situation	Context in one or two sentences
Task	Your specific responsibility
Action	What you did, with technical detail
Result	The outcome, quantified

Practice AI engineer interview questions

FAQs

Is "AI engineer" its own interview track?

At most companies the AI-engineering interview is a software or ML engineer loop with retrieval, serving, and agent questions added. Anthropic, OpenAI, Scale, and Sierra ask their AI infrastructure questions inside SWE and ML rounds, so prepare the standard engineering fundamentals and the AI-specific topics together. Practice both in our question bank.

What's the most common AI engineer interview question?

Some version of "design a batching system so a GPU serves many requests efficiently without breaking latency," and "design an agent/RAG system while controlling token cost." Anthropic and Scale ask these repeatedly. Have the serving framework and RAG/agent framework ready.

Do AI engineer interviews still include LeetCode-style coding?

Less abstract puzzling, more project-style building, often with an AI assistant allowed. You'll still see data-structure fundamentals (LRU cache, key-value stores, graph traversal), especially because the role is rooted in SWE. The difference is the problems look like real features: image pipelines, retry mechanisms, credit systems.

How much ML modeling do I need to know?

Enough to reason about trade-offs and to debug a misbehaving model, which ML-engineer loops test directly (see Anthropic's overconfident-model question). Deep training internals matter more for pure ML roles. Know the difference between teaching a model behavior (fine-tuning) and giving it facts (retrieval).

How do I prepare for an AI engineer interview?

Three weeks at roughly 90 minutes a day covers most loops: week one on fundamentals (RAG, embeddings, serving), week two on system design (batching, agents, cost control), week three on AI-assisted coding and company-specific behavioral prep. Practice out loud against real questions.

Top AI engineer interview questions

AI Engineer Posts

Read our latest blog posts about AI Engineer on Exponent.

45+ AI Engineer Interview Questions & Answers (2026 Guide)

Follow Us