31 tools· updated sep 2026

AI agent architecture diagram: the six layers I actually run

Marek.

  • ai-agents
  • architecture
  • agent-ops

A practical AI agent architecture has six layers: interface and goal, orchestration, context and state, model, tools and runtime, and policy plus observability. The model is only one box. Most production reliability comes from the boundaries around it.

This is the reference diagram I use when a whiteboard has become too vague:

                         +----------------------+
                         | 6. Policy and        |
                         |    observability     |
                         | approvals, traces,   |
                         | evals, budgets       |
                         +----------+-----------+
                                    |
+---------------+        +----------v-----------+
| 1. Interface  | goal   | 2. Orchestration     |
| user, event,  +------->| loop, routing, stop  |
| scheduled job |        | conditions, retries  |
+---------------+        +----+-------------+---+
                              |             |
                     +--------v----+   +----v----------+
                     | 3. Context  |   | 4. Model      |
                     | state, data,|   | inference and |
                     | memory      |   | structured out|
                     +--------+----+   +----+----------+
                              |             |
                              +------v------+
                                     |
                         +-----------v----------+
                         | 5. Tools and runtime |
                         | APIs, MCP, browser,  |
                         | database, queue      |
                         +----------------------+

The arrows are control and data flow, not permission. Every action still passes the policy for that tool and environment.

Layer 1: interface and goal

An agent needs a bounded reason to run. The trigger may be a user request, an event, an alert, or a scheduled job. The interface normalizes that input into a task contract.

I want the contract to state:

  • the goal;
  • allowed resources and tools;
  • completion criteria;
  • time, step, and cost limits;
  • actions that require approval;
  • the result format.

"Improve marketing" is not a useful goal. "Read the last seven days of approved performance sources and propose three anomalies for review" is much easier to operate.

The interface also authenticates the requester and attaches the correct tenant, project, or environment. Authorization should not be inferred later from free-form text.

Layer 2: orchestration

The orchestrator owns the agent loop. It assembles context, calls the model, validates its proposed action, executes an approved tool, records the result, and decides whether to continue.

receive goal
load allowed state
while within step, time, and cost budgets:
  ask model for the next structured action
  validate action against policy and schema
  execute or request approval
  append sanitized result to state
  stop on success, escalation, or terminal failure

Retries, timeouts, idempotency, and stop conditions belong here. They should not depend on the model remembering a sentence that said "do not loop forever."

Anthropic's building effective agents describes simple composable patterns such as routing, parallelization, evaluator-optimizer loops, and autonomous tool use. I start with the simplest pattern that matches the task.

Layer 3: context and state

Context is what the model can see for the current call. State is what the application retains across steps or runs. Memory is one possible source of state, not a synonym for the model's context window.

This layer may contain:

  • the current goal and plan;
  • recent tool results;
  • retrieved documents;
  • conversation history;
  • durable task records;
  • approved user or project preferences.

I assemble context deliberately. Each item gets a source, timestamp, sensitivity level, and relevance reason where practical. Old tool output and untrusted web text should not silently become permanent instructions.

The AI context window guide explains why stuffing every available document into one request is not a memory strategy.

Layer 4: model

The model interprets the current context and proposes the next response or action. I ask for structured output when software must act on it.

{
  "action": "query_metric",
  "arguments": {
    "metric": "registrations",
    "date_range": "previous_7_days"
  },
  "reason": "Confirm whether the alert is visible in the source system"
}

The application validates this object before execution. The reason is useful for review, but it does not grant permission.

Model selection is an architecture decision. A smaller model may handle routing while a stronger model handles difficult analysis. However, adding models also adds failure modes, cost, and evaluation work. I prefer one model until measurements justify a split.

Layer 5: tools and runtime

Tools turn model output into observations and actions. They may call APIs, query a database, use a browser, edit a repository, or communicate through an MCP server.

The runtime wraps those tools with operational controls:

  • typed input schemas;
  • narrow identities and scopes;
  • timeouts and retry rules;
  • idempotency keys for side effects;
  • sandboxing and network restrictions;
  • queues for long-running work;
  • separate read and write paths.

The MCP architecture provides a useful host-client-server boundary for tools and data. It does not replace authorization. My MCP server security checklist covers the controls I add around that boundary.

A tool result is untrusted input when it comes from the web, a document, or another service. The orchestrator sanitizes and labels it before returning it to the model.

Layer 6: policy and observability

I draw this layer across the whole diagram because it is not a final logging box. Policy checks happen before tool exposure, before execution, and before a result leaves the system. Observability records what each layer decided and did.

The policy side includes:

  • identity and authorization;
  • tool and argument allowlists;
  • human approval gates;
  • data classification and redaction;
  • budgets and rate limits;
  • environment boundaries;
  • terminal stop rules.

The observability side includes:

  • task and trace IDs;
  • model and prompt versions;
  • tool calls and sanitized results;
  • approvals and policy decisions;
  • latency, tokens, cost, and retries;
  • outcome and evaluation scores.

OpenTelemetry defines emerging generative AI semantic conventions for common attributes. I still design the event schema around the questions operators must answer, not around whichever dashboard is available.

See AI agent monitoring for the live loop and LLM evaluation for pre-production tests.

The three flows I trace separately

One architecture diagram becomes clearer when I distinguish three flows.

Control flow

Who decides the next step? The orchestrator owns the loop. The model proposes. Policy permits, denies, or asks for approval.

Data flow

What information enters model context, where did it come from, and where is it retained? This is where privacy, freshness, prompt injection, and context quality appear.

Action flow

Which external system changes, under which identity, with what idempotency and rollback path? A read query and a production deployment should not share an undifferentiated tool channel.

Tracing these flows separately makes hidden trust assumptions visible.

Single agent or multiple agents?

I start with one orchestrator and a small tool set. A multi-agent design is justified when responsibilities have genuinely different contexts, permissions, scaling needs, or evaluation criteria.

Microsoft's agent orchestration patterns describe sequential, concurrent, handoff, group chat, and manager patterns. Each solves a different coordination problem. None removes the need for shared state, error handling, authorization, or a terminal condition.

Multiple named agents can make a demo feel organized while hiding extra latency and unclear ownership. Before splitting, I ask whether a deterministic function, parallel tool call, or smaller context would solve the same problem.

For concrete harness comparisons, see the Muse agent harness and DeepSeek harness field notes.

Failure paths belong in the diagram

The happy path is only half the architecture. I draw explicit exits for:

  • invalid model output;
  • tool timeout or rate limit;
  • stale or missing data;
  • denied policy decision;
  • required human approval;
  • exceeded step, time, or cost budget;
  • partial side effect;
  • operator cancellation.

Each path needs an owner and a final state. Retrying forever is not resilience. A safe agent can say "insufficient evidence," preserve the trace, and stop.

Anthropic's work on trustworthy agents frames legibility and control as core challenges for capable systems. In architecture terms, that means the operator must be able to inspect the state, understand the action path, and interrupt it.

What I deploy as one versioned unit

I release more than model code. The artifact should identify the behavior that was tested:

{
  "agent_version": "triage-12",
  "prompt_version": "triage-system-8",
  "model": "provider-model-version",
  "tool_registry": "ops-readonly-5",
  "policy_bundle": "ops-policy-7",
  "context_builder": "context-9",
  "eval_suite": "triage-eval-11"
}

This lets me connect an observed trace to the exact prompt, model, tools, policy, and test suite. The AI agent deployment guide covers how I move that unit through sandbox, staging, shadow, canary, and production.

Frequently asked questions

What are the main components of an AI agent architecture?

The main components are a goal interface, orchestrator, context and state layer, model, tools and runtime, plus policy and observability across the system.

Does an AI agent need long-term memory?

No. Many useful agents need only task state and recent tool results. Add durable memory when the product has a clear retrieval, privacy, correction, and deletion policy.

Where should human approval sit?

Approval should sit in the action path before a sensitive tool executes. It should be enforced by the orchestrator or tool gateway, not only requested in the prompt.

Is MCP the architecture for an AI agent?

MCP can standardize connections to tools, resources, and prompts. It is one part of the tool layer, not the complete agent architecture.

When should I use a multi-agent architecture?

Use it when separate roles need different permissions, contexts, scaling, or independent evaluation. For a narrow task, one agent with clear tools is usually easier to operate.