Design an AI-Powered Customer Support System

You're asked to design an AI agent that handles customer support. A customer describes a problem in their own words, and the system answers from the company's documentation, looks up their account, and where appropriate takes an action on their behalf, such as issuing a refund, resetting a password, or changing a subscription. When it can't resolve the issue, it hands off to a human with the context already gathered.

This is an agentic RAG system: the model answers from documents retrieved at question time and can also take actions. Wrong actions can cost money, and the knowledge corpus changes constantly. Let's spend most of our time on retrieval quality, tool safety, and escalation rather than raw request throughput.

Clarify the requirements

• Can the agent take actions, or only answer? Ask this first. An answering-only system is primarily retrieval, while an acting system needs guardrails, idempotency, and approval gates.
• What's the escalation path? There's always a human somewhere. How the handoff works, and what the human receives, is part of the system rather than outside it.
• What's the knowledge source? Help center articles, internal runbooks, past resolved tickets, and product documentation. Past tickets are the most valuable and the messiest.
• What accuracy bar? A wrong refund is a different failure from a wrong answer. Ask what the tolerance is, because it sets where the guardrails go.
• Which channels? Chat, email, and voice have different latency expectations. Chat is the default assumption.

Assume: the agent can act within limits, escalates to humans, draws on docs and past tickets, and runs in chat.

Back-of-envelope numbers

• Volume: $100\text{k conversations/day} \approx 1.2/\text{sec}$ average, peaking around $10/\text{sec}$, which is a small request rate
• Turns: $6 \text{ turns/conversation}$, each with a retrieval call and one or more model calls
• Model cost: $100\text{k} \times 6 \text{ turns} \times \$0.02 = \$12\text{k/day}$, against a human agent cost of roughly $\$5$ per contact
• Knowledge base: $50\text{k documents} \times 5 \text{ chunks} = 250\text{k chunks}$, a small vector index by any measure
• Deflection value: at $60\%$ resolution, $60\text{k} \times \$5 = \$300\text{k/day}$ of human cost avoided

Put the knowledge-base size and deflection value on the whiteboard. The request rate and index are small, while the value depends on maintaining quality. We should optimize for correct resolutions rather than raw throughput.

High-level architecture

Components

1. Agent orchestrator. Owns the loop of decide, retrieve or act, observe, repeat, and enforces its budget.
2. Retrieval service. Turns a customer's phrasing into the right passages from the knowledge base.
3. Vector and keyword index. Hybrid search over chunked documents and past resolutions.
4. Policy layer. Deterministic code deciding which actions the agent may take unaided.
5. Account tools. The real APIs for refunds, subscription changes, and password resets.
6. Escalation. Handoff to a human, carrying the transcript and what was already established.
7. Conversation store. Transcript, tool calls, and retrieved context, for audit and for evaluation.
8. Knowledge ingest. The pipeline that keeps the index current as documentation changes.

Deep dive 1: Retrieval

For the agent to answer accurately, it has to find the right support documents. A basic RAG implementation has three steps: embed the customer's question, find the nearest chunks by vector similarity, and put them in the prompt.

For production, let's add four retrieval refinements so the model receives the right passages.

Chunk on the document's structure. A whole document is too vague to match against; a single sentence loses the context that made it meaningful. Splitting on sections and headings uses boundaries the author already drew. Overlap the chunks slightly so a passage split across a boundary is still retrievable, and prepend the document title and heading to each chunk so an isolated paragraph still says what it's about.

Run keyword search alongside vector search. Embeddings match on meaning, so they find "can't sign in" when the customer wrote "locked out." They're weak on exact strings, which is a problem when the customer hands you an error code or an order number. Running BM25 next to vector search and fusing the two ranked lists gets both behaviors.

Rerank before you generate. Retrieval should optimize for recall: pull fifty candidates so the right one is probably among them. Then score those fifty with a cross-encoder and keep the best five. Cross-encoders are far more accurate than embedding similarity and far too slow for the whole corpus, which is why they belong in a second stage.

Filter by metadata before searching, not after. Narrow to the customer's product version, locale, and plan first. A passage about a plan they aren't on produces an answer that is fluent, confident, and wrong.

Deep dive 2: Building and maintaining the vector store

The index the agent searches is itself a pipeline, and once the system is live that pipeline is where most of the operational work is.

Re-embed on change, not on a schedule. Documentation edits should flow through as they happen, since a stale answer about a policy that changed last week is exactly the failure that erodes trust. Content-hash each chunk so unchanged chunks aren't re-embedded, since most edits touch a small part of a document.

Past tickets are the most valuable and hardest source. Resolved conversations contain the answers to questions the documentation never anticipated. They also contain customer names, order numbers, and card details, so they need PII scrubbing before indexing, and they need filtering for resolution quality, because indexing a ticket where the customer left angry teaches the agent to give that answer.

Re-embedding the whole corpus on a model change is a real operation. Swapping the embedding model invalidates every vector, and query and document embeddings must come from the same model or retrieval silently degrades. Building the new index alongside the old and cutting over atomically is the same versioned-publish pattern used throughout batch processing and data pipelines.

Evaluate retrieval separately from generation. This is the point most candidates miss, and it's the one that makes the system improvable. A golden set of real questions with the passages that should have been retrieved gives you recall@k on retrieval alone. When quality regresses, that number tells you immediately whether the problem is finding the right content or using it, and those are two entirely different fixes. Evaluating only end-to-end answer quality leaves you guessing.

Deep dive 3: Taking actions safely

The moment the agent can issue a refund, this stops being an information system. The governing principle is that the model proposes and deterministic code disposes. The agent decides what it wants to do, and non-model code decides whether it happens.

Tier actions by reversibility, which is the axis that actually matters:

• Read-only. Order status, account details, shipment tracking. Available freely.
• Reversible writes. Updating a preference, resending an email. Available with logging.
• Consequential writes. Refunds, cancellations, plan changes. This is limited by policy: a refund under a threshold within the return window on a verified account proceeds; anything outside those bounds goes to a human.
• Irreversible or high-value. Always human-approved.

The thresholds live in the policy layer as configuration, not in the prompt. A prompt instruction not to refund above $100 is a suggestion; a check in the policy layer is a rule.

Idempotency is not optional. The orchestrator can crash after calling issue_refund and before recording the result, and the retry double-refunds. Every mutating tool call carries a key derived from the conversation and step, and the tool is idempotent against it, exactly as described in transactional workflows.

Authorization belongs to the customer's session, not the agent. Tools execute with the authenticated customer's permissions. A prompt injection that asks for someone else's order then fails in deterministic authorization code rather than relying on model judgment.

Put a hard limit on the loop. Cap the number of steps, the tokens spent, and the wall-clock time, and decide what happens when a limit is hit. Here the answer is to escalate to a human, since this system always has that fallback available.

Deep dive 4: escalation, and knowing when to stop

The measure of this system isn't how many conversations it handles, it's how many it handles well, and a system that never escalates is worse than one that escalates often.

Escalate on explicit triggers, not on model discretion alone. A customer asking for a human, sentiment turning negative, a policy boundary reached, the loop exhausting its budget, repeated failure to resolve, and any topic on a designated list covering billing disputes, account security, or legal should route out immediately.

Hand the human everything the agent already gathered: the transcript, the passages it retrieved, the actions it attempted, and what it managed to establish. Making the customer repeat all of it is the failure that makes people hate these systems, and avoiding it accounts for most of the perceived quality difference between a good deployment and a bad one.

Design for graceful uncertainty. When retrieval returns nothing relevant, the agent should say it doesn't know and escalate rather than generating a plausible answer from the model's parameters. Making that the default behavior, where no relevant passages means no answer, turns hallucination from a model problem into a routing decision.

Measure resolution, not deflection. Deflection counts conversations that didn't reach a human, which improves when the agent is unhelpful enough that customers give up. Resolution counts problems actually solved, verified by follow-up contact rate and satisfaction. Choosing the honest metric is a genuine judgment signal, because the dishonest one is easier to move and it's what a poorly designed program optimizes.

Common pitfalls

• Textbook RAG and nothing more. Embed, retrieve by vector similarity, generate. Pure vector search misses the exact strings customers give you, and unranked results waste the prompt.
• Guardrails written into the prompt. Policy in a prompt is a suggestion; policy in code is a rule.
• Agent-level authorization. Tools must run with the customer's permissions, or prompt injection becomes data access.
• Answering when retrieval found nothing. The model fills the gap from parameters, fluently and incorrectly.
• Optimizing for deflection. An unhelpful agent deflects beautifully.

Leveling signals

Related lessons

Design ChatGPTHard

Serve a multi-turn conversational product backed by a large language model.

Design an LLM Query Batching SystemHard

Batch inference requests to maximize GPU utilization while users wait synchronously.