AI agent deployment: the release path I use before production

AI agent deployment is the point where a convincing demo meets credentials, real users, bad inputs and consequences. The model call is rarely the difficult part. The release path is. I use five stages: sandbox, staging, shadow traffic, a small canary and production. Each stage answers a different question, and none is allowed to borrow the permissions of the next one.

• Sandbox asks whether the agent can complete the task. Nothing valuable is writable and every input is disposable.
• Staging asks whether the whole system works. Real schemas, fake or copied data, production-shaped limits and no production authority.
• Shadow asks what the agent would have done. Real inputs, recorded outputs, no side effects.
• Canary asks whether a small slice is safe. Narrow traffic, narrow scope and a rollback trigger written before release.
• Production asks whether the operating loop is honest. Traces, review, budgets and an owner who can stop it.

This is the release companion to my 12-gate production checklist. That post asks whether an agent is ready. This one shows how I move it without turning the first real user into the final test case.

What AI agent deployment actually includes

Deploying an AI agent means releasing an application, a model configuration, a prompt, a tool surface, credentials, state and operating controls as one versioned unit. Changing any one of them can change behaviour. A prompt edit can redirect tool use. A model upgrade can change how aggressively the agent acts. A harmless new tool can widen the set of reachable data. Treating only the container image as the release leaves most of the system unversioned.

Anthropic makes a useful distinction between fixed workflows and agents that direct their own process and tool use. It also recommends sandbox testing and stopping conditions for autonomous loops: https://www.anthropic.com/engineering/building-effective-agents

My smallest useful release record looks like this:

release: agent-2026-08-26.1
model: pinned-model-version
prompt_sha256: 7a31...
tools:
  - search.read@3
  - report.write@2
credential_profile: report-writer-prod
eval_set: report-agent-v14
rollback_to: agent-2026-08-19.2
owner: operations

The exact format does not matter. The ability to answer "what changed?" does. OpenAI's production checklist makes the same operational point from another direction: plan for scaling, latency, security, monitoring and cost before going live, not after the first incident: https://developers.openai.com/api/docs/guides/deployment-checklist

Stage one: sandbox the task, not just the code

A sandbox is not a staging account with a warning in the prompt. It is an environment where the agent cannot cause a production side effect even if the model ignores every instruction. I start with fixtures, temporary files, fake recipients and credentials that cannot reach production resources.

For terminal workflows, use the Claude Code sandbox boundary checks to probe permitted files and network destinations before the real task.

This stage answers three questions. Can the agent finish representative tasks? Can it recover from tool errors? Does it stop when the task becomes ambiguous? The third question is the one demos usually skip. A capable agent can make a bad guess faster than an incapable one.

Every sandbox run should leave a full trajectory: input, prompt version, model, tool calls, tool results, output and stop reason. OpenTelemetry now defines GenAI attributes for agent identity, conversations, messages and evaluation results, while warning that recorded prompts may contain sensitive data: https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/

Record enough to reproduce behaviour, then redact or exclude secrets and personal data deliberately. "Log everything" is not permission to build a second data leak.

Stage two: make staging production-shaped

Staging should use the same tool schemas, rate-limit handling, queue semantics and identity boundaries as production. It should not use the same secrets. If the agent needs a database, give staging its own role and dataset. If it sends messages, route them to a sink. If it writes files, give it an isolated prefix.

The failure I am looking for here is integration drift. A tool renamed one field. A downstream API returns 429. A session expires halfway through a task. The model returns valid prose where the next component expects JSON. These are distributed systems failures with a probabilistic component attached.

Microsoft's agent orchestration guidance calls out timeouts, retries, graceful degradation, output validation, circuit breakers and checkpoints. It also warns that errors in an early stage can cascade into every later agent: https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/ai-agent-design-patterns

That is why I validate between steps. The model does not get to decide that its own malformed output is close enough.

Stage three: shadow real traffic without acting

Shadow mode feeds production-shaped inputs through the candidate and records what it would do, but blocks side effects. Reads can still be sensitive, so shadow is not automatically low risk. The difference is that a wrong conclusion becomes a review item rather than an email, database update or payment.

For each shadow run I compare the candidate against the current system or a human decision. I care about disagreements, not only a single average score. Which inputs trigger extra tool calls? Which records would change? Where does the new version abstain more or less often? A release with the same aggregate pass rate can still introduce one unacceptable failure class.

Shadow mode also exposes cost and latency on the real input distribution. A test set rarely contains the giant attachment, ten-tool detour or repeated retry that defines the expensive tail.

Stage four: canary with a written rollback trigger

A canary release gives the candidate a small, identifiable slice of real work. The important number is not the percentage. It is the blast radius. One percent of password resets is not equivalent to one percent of read-only summaries.

Before the canary starts, I write the stop conditions:

rollback if:
- any unauthorized tool call is attempted
- task success drops below the approved baseline
- p95 cost per completed task exceeds the budget
- human reversals exceed the agreed threshold
- traces or review data become unavailable

Kubernetes Deployments popularised a useful operational property: versioned rollouts with status and rollback instead of replacing everything at once. The mechanism is platform-specific, but the principle transfers directly: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/

Rollback has to restore the whole agent release, not only the code image. Restore the prompt, model pin, tool definitions and credential profile together. If state was mutated under the candidate, code rollback may not undo the damage. For high-impact writes, use reversible actions, append-only events or a human approval before commit.

Stage five: production begins the monitoring job

A successful canary is evidence, not immunity. Inputs change, providers change, tools change and users discover paths no test set contained. Production therefore needs full traces, deterministic health checks, sampled quality review, cost budgets and an explicit decision trail.

My agent monitoring loop starts with trajectories, routes suspicious runs into a human queue, samples the rest with a cheap judge and turns reviewed failures into regression tests. That last step is what makes the system improve. A dashboard that only reports failure leaves the same trap open.

The owner also needs a kill path that does not depend on the agent being healthy. Disable the trigger, revoke the credential, stop the queue consumer or route all actions to approval. "Tell the agent to stop" is not an operating control.

Frequently asked questions

How do you deploy an AI agent safely?

Move it through sandbox, production-shaped staging, shadow traffic and a small canary before full release. Keep permissions narrow at every stage, version the model, prompt and tools together, and define rollback triggers before the canary.

What is the difference between AI agent deployment and model deployment?

Model deployment serves a model endpoint. AI agent deployment releases the whole application around it, including prompts, tools, credentials, state, policies, evaluation data, traces and rollback controls.

Should an AI agent use production data in staging?

Prefer synthetic or sanitized data with the same schema and edge cases. If a test requires production data, use an isolated read-only path, minimise the fields and apply the same privacy and retention controls as production.

What should trigger an AI agent rollback?

Rollback on unauthorized action attempts, quality regression, budget breach, unavailable traces or a rise in human reversals. The exact thresholds depend on impact, but they must be measurable and written before release.

The release rule I keep

An agent is not deployed when the endpoint responds. It is deployed when I can name the version, show what it saw, bound what it can do, compare it with the last version and stop it without asking for cooperation. The five stages are simply a way to earn those answers in increasing order of consequence.

The next step after release is not another feature. It is using the saved time to make the regression net wider, which is why my AI time dividend goes to tests.