Retrieval-Augmented Generation

What happens when you ask an AI model about today's news or your company's internal documentation? In both cases, the model is missing the information it needs: today's events happened after training, and your private documents were never part of its training data. Retrieval-augmented generation (RAG) closes that gap by giving the model relevant source material at question time. You'll see it in nearly every kind of AI product, including search summaries, support agents, coding assistants, and agents with long-term memory.

With RAG, we fetch the relevant passages when the user asks a question and add them to the context. The model can then answer from what it just read rather than what it vaguely remembers.

During an interview, look for prompts where the model needs to answer questions about dynamic or proprietary data.

The core idea

A model's knowledge is frozen at training time, so it can't access private data or anything newer than its cutoff. Ask about your internal documentation and the model may refuse or hallucinate an answer. You could fine-tune it on that documentation, but fine-tuning is expensive, slow to update, and still unreliable at recalling and citing specific facts.

RAG takes a different approach. Before the model answers, a retrieval system searches a knowledge base for relevant passages and adds them to the prompt. The model now has an easier task: read these passages and answer the question. The technique comes from a 2020 Facebook AI Research paper by Lewis et al. titled "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." The name describes the pipeline: retrieve, augment, generate.

Think of an LLM without RAG as a chef cooking from memory and improvising when they don't recognize a dish. RAG hands the chef the recipe before they start cooking.

We'll use one example throughout the lesson: design a chat agent that answers questions about your company's support policies. This prompt gives us a practical way to examine every part of the pattern.

Start by separating the system into two paths that run at different times:

Retrieval path (online): question → retrieve top-k chunks → assemble prompt → generate answer. Here, top-k means the top few matching passages. You'll also hear this called the query path. It's latency-sensitive and runs on every request, while a user waits.

Storage path (offline): documents → chunk → embed → index. You'll also hear this called the ingest path. It's a batch pipeline that runs on a schedule or on document change, typically well before any query arrives.

Design these paths separately. They have different latency budgets, scaling behavior, and failure modes. We'll start with the two forms of the retrieval path, then work through the storage path one layer at a time.

Traditional RAG: retrieve once, answer once

Traditional RAG, sometimes called naive or standard RAG, uses a fixed pipeline: retrieve once, answer once.

Suppose a user asks the support agent, "Can I return a jacket I bought 45 days ago?" The pipeline searches the knowledge base and retrieves the top few matching passages, such as the return policy and its holiday exception. It adds those passages to the prompt beside the question, and the model answers from them.

The pipeline controls retrieval here. The model does not decide what to search or whether to try again; it reads whatever the retriever provides. In our kitchen analogy, we hand the chef one recipe and ask them to follow it. This makes traditional RAG fast, cheap, predictable, and easy to debug, so it should be your default for single-hop questions.

Traditional RAG has two common failure modes:

• Multi-step questions. "Does my order from March qualify under the new return policy?" requires fetching the order first, then checking the policy against it. One retrieval can't do both.
• Bad first retrieval. If the top-k chunks come back off-topic or empty, the model answers from bad context anyway. There's no second chance built into the pipe.

In our analogy, the chef is stuck when the recipe calls for a missing ingredient because they cannot leave the kitchen to find it.

Agentic RAG: the model drives the search

With agentic RAG, the LLM controls retrieval. Search becomes a tool the model can call as it works. The model decides what to search for, examines the results, and determines whether it has enough information to answer. If it does not, it can refine the query or search another source, such as the policy documentation, order database, or past support tickets.

The chef still receives a recipe, but now they can visit the grocery store too. If an ingredient is out of stock, they can check another aisle or make a substitution.

Let's run a harder question through the loop. The agent first queries the order system to find the purchase date. It then searches the policy documentation with that date and uses both results to answer. The first search informs the second.

This has the same tradeoffs as any agent loop, which we cover in P-09. Each iteration adds latency and token cost, so the system needs a termination condition. During an interview, set a maximum number of retrieval rounds and define what happens when the agent reaches it, such as answering with the information it has.

Start with traditional RAG. Move to agentic RAG when questions require multiple steps, span several sources, or need a second attempt after weak retrieval. A simple FAQ workload usually does not need an agent loop.

The storage path: building the knowledge base

Both approaches depend on a knowledge base that can answer a search. Let's build that knowledge base one layer at a time. This work happens offline, before the user submits a query. In the kitchen analogy, we're stocking the pantry before service.

Layer one: chunking. We do not index each document as one large unit because the model needs the relevant paragraph, not the forty-page manual around it. Split documents into small, self-contained passages. If a chunk is too small, it loses context. If it is too large, it consumes the prompt budget and dilutes the relevant information.

Layer two: embeddings. An embedding model converts each chunk into a vector, which is a list of a few hundred to a few thousand numbers. Text with similar meaning lands close together in this vector space. "Money back" and "refund policy" share no words, but their vectors become neighbors.

Layer three: the vector index. Store those vectors in an index designed to find the nearest neighbors of a new point quickly, usually with an approximate nearest neighbor structure such as HNSW. At query time, embed the question in the same vector space. The closest chunks become the search results.

Before moving on, address three operational questions:

• Freshness. When a policy document changes, how quickly does the update reach the index? Treat this as a batch pipeline problem involving scheduling, change detection, and reindexing cost.
• Access control. Filter results according to the user's permissions at query time, not when you build the index.
• Scope. If the corpus is small and stable, you may not need this machinery. Put the source material directly in the context window instead.

One more layer: embeddings alone aren't enough

Semantic search works well when the wording changes but the meaning stays similar. It is less reliable for exact identifiers, product codes, and names. That's where lexical search helps. An inverted index, the approach behind Elasticsearch and search engines since the 1990s, can use a scoring method such as BM25 to find ERR_CONN_RESET every time. But it may miss "how do I get my money back" when the relevant page is titled "refund policy."

The two approaches fail in different ways, so production systems often use hybrid retrieval. Query both indexes, merge their candidates, and then use a reranker model to reorder the combined list before choosing the final top-k results. Querying both improves recall, while the reranker improves precision.

Tip: Don't stop at "use a vector database." Compare lexical and semantic search, explain where each one fails, and combine them with a reranking step when the workload needs both.

Semantic caching

LLM calls are the expensive part of the retrieval path, and user questions are often repetitive. Thousands of people may ask "how do I reset my password" with slightly different wording, which means an exact-match cache will miss most of those requests.

A semantic cache stores each answered question's embedding alongside its answer. For a new query, create an embedding and check whether a cached question falls within a chosen similarity threshold. On a hit, return the stored answer in milliseconds at near-zero cost. On a miss, run the full RAG pipeline and cache its result.

Watch for two risks. A loose threshold can cause a false hit and return an answer to a subtly different question. The cache can also become stale when the source documentation changes, so its invalidation policy needs to connect to the storage path.

Leveling signals

• A mid-level answer explains the traditional pipeline: chunk, embed, index, retrieve, prompt, and generate. It also explains why RAG is a better fit than fine-tuning for fresh, factual data.

• A senior answer separates the retrieval and storage paths, discusses chunk size, compares lexical and semantic retrieval, and proposes a hybrid when appropriate. It also covers freshness, knows when to move from traditional to agentic RAG, and asks how we'll evaluate whether retrieval finds the right chunks.

• A staff-level answer treats RAG as a combination of patterns: a batch pipeline feeds a search system, which may be used by an agent loop. It uses semantic caching to control cost and enforces access at retrieval time according to the asking user's permissions. It also recognizes when RAG is unnecessary because a small, stable corpus can fit in the context window.

Practice this pattern

• Design an AI-Powered Customer Support System: the agent + RAG hybrid, end to end.
• Design Typeahead / Search Autocomplete: the retrieval layer's read-path discipline.
• Design a Web Crawler: the storage side at scale.

In one sentence

RAG retrieves relevant passages from a knowledge base at question time and adds them to the model's prompt. Traditional RAG searches once through a fixed pipeline, while agentic RAG lets the model search in a loop. Hybrid retrieval combines lexical and semantic search, and semantic caching reduces the cost of repeated questions.