Agentic AI Architectures
You ask an AI assistant to research a topic, update a customer account, or modify a codebase. How does it decide which tools to use and what to do after each result? An agent runs a model in a loop: plan, call a tool, observe the result, and repeat until the task is complete.
Agents build on familiar system design patterns. An asynchronous worker (async jobs and workers) can own the loop and its state, while a real-time channel (real-time and collaborative systems) streams progress to the user. The new part is an LLM making control-flow decisions: which step comes next, which function to call, and when to stop. We need to bound those decisions with context management, tool permissions, guardrails, evaluation, and cost controls.
The core idea
A plain LLM call is a function: prompt in, text out. An agent turns that into a control loop. The runtime sends the model the task plus the conversation so far plus tool definitions; the model responds with either an answer or a tool call; the runtime executes the tool, appends the observation, and calls the model again. The loop ends when the model produces a final answer, or when the runtime stops it.
Most agentic systems have four components:
- Harness / orchestrator. The program that owns the loop, decides when to call the model, executes the tool the model asked for, and decides when to stop.
- Tools. Search, retrieval, database queries, code execution, and external APIs, each with a schema the model sees and a set of permissions the model doesn't get to change.
- State. Conversation history, intermediate results, and checkpoints, held by the orchestrator rather than inferred from the transcript.
- Model gateway. The layer that routes calls to model providers, with retries, fallbacks between models, and usage accounting per task and per tenant.
Two properties distinguish this pattern. The model decides the step count at runtime, so cost and latency remain unbounded until the runtime sets limits. The model is also nondeterministic, so the same input may produce different tool calls across runs. Testing and debugging must account for that variation.
Deep dive: control flow
The loop alone is incomplete. Add the runtime controls that make it safe and operable.
Termination and budgets. Give every loop a hard step cap, token budget, wall-clock timeout, and spend limit per task and tenant. Without those limits, a model may retry a failing tool forever or two agents may pass a task back and forth. The runtime, not the model, must enforce termination. Define what happens at the limit: fail the task, summarize progress, or escalate to a human.
Idempotent side effects. Suppose the runtime calls refund_customer and crashes before recording the result. Retrying the step could refund the customer twice. Mutating tools need the same controls as async jobs and workers: idempotency keys derived from the task and step, upserts instead of inserts, and a status check before re-execution. Reads can usually retry freely; writes need keys.
Guardrails. Control what the agent can do, not only what it can say. Validate tool parameters in deterministic code, restrict tools with allowlists and permissions, and require human confirmation for irreversible actions. The model proposes an action; the runtime decides whether to execute it.
Deep dive: context management
The model sees only its context window. In a long-running task, conversation turns, tool results, and retrieved documents all compete for that limited space. Treat context as a managed resource rather than appending every result indefinitely.
Use three techniques together:
- Truncation and summarization. Summarize the transcript so far and keep only recent turns verbatim, so the loop's context grows sublinearly with its step count.
- Retrieval instead of stuffing. Store documents outside the prompt and fetch only what the current step needs, which is cheaper and usually more accurate than a giant prompt.
- Structured state outside the context. The orchestrator tracks the plan and intermediate results in its own data structures, and the prompt carries only what the next decision requires.
RAG, or retrieval-augmented generation, fits here as a retrieval tool. An ingest pipeline built with batch processing and data pipelines chunks, embeds, and indexes documents. At answer time, the agent retrieves relevant chunks and cites them. This grounds answers in current or private data without retraining. It also lets us measure whether retrieval returned the correct evidence rather than treating every wrong answer as an undefined model failure.
Deep dive: serving, latency, and cost
Model calls dominate latency and cost. Optimize for both response time and cost per completed task.
- Stream progress. Send tokens to the user over SSE as they are generated, using the one-way push from real-time and collaborative systems. Ten seconds of visible progress feels different from ten seconds of silence, even when total runtime is unchanged.
- Route by difficulty. Let a smaller model classify and handle easy turns, then escalate harder ones to a larger model. Since most traffic is easy and the per-token cost differs by an order of magnitude or more, routing can substantially reduce unit cost.
- Cache what repeats. Exact-match and semantic caches for common questions, plus prompt-prefix caching so a long system prompt isn't reprocessed on every call.
- Degrade deliberately. When the primary provider is slow or unavailable, fall back to another model or a scripted flow. Define which capabilities or quality guarantees change during that fallback.
Behind the gateway sits the batching and queueing layer that feeds the hardware, which is its own design question. Check out Design an LLM Query Batching System.
Deep dive: evaluation and observability
Because model behavior varies across runs, evaluation is part of the production system rather than a one-time test. Build an evaluation strategy before changing prompts or models in production.
Every AI eval system needs:
- A golden set. A fixed collection of representative tasks with expected outcomes or grading criteria, run on every prompt or model change. Treat a prompt edit as a deploy, because that's what it is.
- A scoring method. Exact match where the answer is checkable, and LLM-as-judge where it isn't, guarded by spot-checking a sample of judgments against human agreement so you know the judge itself is calibrated.
- Agent-specific grading. For a multi-step loop, score the trajectory as well as the final answer (i.e. whether the right tools were called in a reasonable order, and within budget).
- Online monitoring. Resolution rate, escalation rate, user feedback, cost per task, and p95 latency per step, measured in production where the real input distribution lives.
- Full traces. Every model call, tool call, and observation recorded for each loop, so a bad outcome can be replayed and diagnosed rather than guessed at.
Regression evals play a role similar to unit tests: they make prompt, model, and tool changes safer.
When to use it, and when not to
Use an agent when a task requires several tool-driven steps, the sequence cannot be fully enumerated in advance, or an open-ended conversation must reason over private data and take actions.
Use a deterministic workflow when the steps are known. A pipeline with one or two LLM calls for classification, extraction, or summarization is cheaper, faster, and easier to test. If the task only retrieves and synthesizes information, RAG without a loop may be enough. Keep the agentic portion as small as the product allows and use deterministic code around it.
Common pitfalls
- No termination budget. One confused loop becomes a five-figure bill.
- Non-idempotent tool calls. A retry double-refunds the customer.
- Irreversible actions with no deterministic check. That's a guardrail failure, not a model failure.
- Stuffing everything into the context. Retrieval beats a giant prompt on both quality and cost.
- No evals. Every prompt change ships untested.
Leveling signals
Practice this pattern
Answer customer questions from company data and take actions on their accounts.
Serve a multi-turn chat product backed by a large language model.
Answer questions over a private document corpus, with citations.
Run a coding agent that reads and edits files and executes commands.