What should block a deployment of an agent when the change is only a prompt, tool description, model version, or policy configuration? Design a CI/CD process that covers replay tests, safety tests, canaries, rollback, and evaluator drift.

How to think about it

I would block the deployment on a critical safety failure, unauthorized tool call, broken tool contract, material replay regression, or a canary breach of latency, cost, or business SLOs. The fact that the diff is “only a prompt” or “only a model version” changes the test mix, not the release bar.

Why a non-code change can break production

An agent is a policy-driven program whose control flow is partly generated at runtime. The prompt influences which action the model chooses. A tool description influences how it understands an action. The model version changes the probability of every action. A policy configuration changes what actions are allowed.

The production behavior is therefore the combination of all four, plus the tool implementations and surrounding orchestration. A one-line description change from “refund an order” to “refund an order when appropriate” can make the model call the refund tool in cases that previously triggered clarification. The diff is tiny. The blast radius is not.

The key distinction is between advice and enforcement. A prompt can tell the model not to refund more than 500 dollars. It cannot enforce that limit. The refund service must check the amount, customer identity, tenant, authorization, and approval state at the point of action. Tool descriptions help the model select and use tools; they are not an access-control system.

That gives me two kinds of release gates:

• Hard safety gates: zero tolerance for unauthorized data access, policy-prohibited actions, secrets in output, cross-tenant access, or malformed high-impact tool calls.
• Quality and operational gates: bounded regression in task success, latency, cost, escalation rate, and user-visible error rate.

A quality regression might require review. A model attempting to email one customer’s invoice to another customer should block immediately.

Package behavior as a release artifact

I would build one immutable release bundle containing:

• the prompt and prompt-template version;
• model provider and model version;
• tool names, descriptions, input schemas, and output schemas;
• policy configuration and policy-engine version;
• orchestration code version;
• evaluator versions;
• replay-set and safety-set identifiers;
• dependency versions and configuration hashes.

The bundle gets a release ID. The evaluator and dataset versions matter because a score without its measuring instrument is not reproducible.

The deployment system should compare a candidate bundle with the currently serving bundle. It should record structured events rather than judging only final prose:

• selected tool and arguments;
• policy decisions and approvals;
• retrieved document identifiers;
• final answer classification;
• latency, token usage, and estimated cost;
• retries, fallback models, and human escalations.

For privacy, production traces used in replay need redaction and access controls. A replay corpus is a test fixture, not a convenient second customer database.

A concrete example: a refund agent

Imagine Northstar, a support agent with three tools:

• lookup_order
• issue_refund
• send_email

The business rule says that refunds above 500 dollars require human approval. The agent may inspect only the authenticated customer’s orders.

The release candidate changes the system prompt and upgrades the model. The tool implementations have not changed. That is still a release, because the candidate may choose different tools or arguments.

The team replays 6,000 redacted historical cases:

• 4,000 ordinary order and refund requests;
• 1,000 ambiguous cases, such as duplicate charges or missing orders;
• 1,000 adversarial cases, including prompt injection in an order note, requests for another customer’s invoice, and attempts to bypass approval.

The replay compares structured outcomes with the current production bundle. It does not demand identical wording. It asks whether the candidate reached the correct business outcome, used the permitted tools, supplied valid arguments, and escalated when required.

For this service, an example release policy could be:

• zero critical safety failures;
• zero unauthorized tool calls;
• zero refund calls above 500 dollars without the required approval;
• no broken tool-schema calls;
• no more than a 1 percentage-point drop in ordinary-case success;
• no increase in median cost above the agreed budget;
• no latency regression beyond the customer-facing SLO.

Those numbers are service policy, not universal truths. The important part is that they are agreed before looking at the candidate’s score. If only 20 examples cover a rare high-risk path, a green percentage is not convincing evidence. Add cases, run targeted tests, or require human review.

A small gate might look like this:

hard_fail = (
    critical_safety_failures > 0
    or unauthorized_tool_calls > 0
    or schema_failures > 0
    or approval_bypasses > 0
)

if hard_fail:
    raise SystemExit("BLOCK")

if success_drop_percentage_points > 1.0:
    raise SystemExit("BLOCK")

print("ELIGIBLE FOR CANARY")

This is release logic, not a vendor API. In a real system, the measurements would also include sample counts, confidence intervals, and comparison against the previous bundle.

The pipeline

1. Fast static and contract checks. Validate prompt templates, configuration schemas, tool input and output schemas, allowed model identifiers, and policy syntax. Run unit tests for the policy engine. Test that a refund request with an amount of 501 dollars cannot execute without approval, even if the model asks for it perfectly.

2. Replay tests. Run the candidate and baseline against the same versioned corpus. Use multiple runs for stochastic or high-risk cases; a single successful run can hide a one-in-ten failure. Compare structured outcomes, not exact generated text. A harmless sentence change should not fail the build, while a changed recipient in send_email should.

3. Safety tests. Include direct policy tests and adversarial tests. Try prompt injection through retrieved content, tool-output poisoning, malformed arguments, data exfiltration, repeated tool calls, and conflicting instructions. Safety tests should exercise the real authorization boundary. If the test merely checks whether the model politely refuses, it is testing manners, not safety.

4. Human review for ambiguous deltas. A candidate that passes hard gates but changes escalation behavior may still deserve review. Route a stratified sample of changed outcomes to reviewers. Store their labels and reasons; these become future regression cases.

5. Canary release. Send a small, controlled slice to the candidate, perhaps 1 percent of eligible traffic, but do not use elapsed time as the only requirement. Require a minimum number of relevant sessions or high-risk actions. For write-capable agents, use a tenant allowlist, approval mode, transaction limits, or a side-effect sandbox. “Canary” must not mean letting an unproven agent issue real refunds at full authority.

Monitor safety events, tool-call denial rate, success proxy, escalation rate, latency, token cost, and customer complaints. A single critical safety event should halt and usually roll back. A modest latency increase may trigger investigation rather than an emergency rollback, depending on the SLO.

6. Progressive rollout and rollback. Increase traffic only after each gate passes. Keep the entire previous bundle available behind the router, not just the old prompt. Rollback should be a configuration change that restores the previous known-good bundle, not a rebuild performed during the 3 a.m. page.

Also make side effects idempotent, meaning repeating the same request does not create a second refund or duplicate email. Rollback prevents new damage; it cannot unsend an email or reverse every already-issued payment.

The senior nuance: evaluators can rot

An evaluator is the mechanism that decides whether behavior is good. It may be a deterministic rule, a human label, or another model judging an answer. Evaluators drift when the rubric, judge model, data distribution, or business policy changes.

I would version evaluators just like agent bundles and pin the evaluator version used by the deployment gate. I would maintain an anchor set with human-reviewed labels and run it whenever the evaluator changes. Track agreement by risk category, not just one overall score: a judge can look stable overall while becoming worse at detecting prompt injection.

If evaluator-human disagreement rises, freeze automated promotion and review the labels and rubric. Never “fix” a failing release by quietly changing the evaluator until the score turns green. That is not CI/CD. That is moving the speedometer.

The trade-off is release speed. Strict zero-failure gates can block harmless improvements because tests are noisy or labels are incomplete. The answer is not to weaken safety gates. Improve the test set, separate critical from noncritical failures, use confidence-aware thresholds, and provide a reviewed exception process with an expiry date and named owner.

What they’ll ask next

“Would every prompt change need the full process?”
Every change needs the same safety boundary, but test depth can be risk-based. A wording-only change may use fast replay and targeted checks; a change affecting tool selection, permissions, or high-impact actions gets the full suite and a canary.

“How do you test a new model when historical traces are biased toward the old model?”
Combine historical replays with synthetic edge cases, production-shadow inputs, adversarial cases, and a human-reviewed sample. Historical data measures continuity; it does not prove coverage of newly introduced behavior.

“What if the canary looks good but the evaluator misses a failure?”
Use independent signals: authorization logs, policy-denial logs, user reports, sampled human review, and business metrics. The evaluator is one sensor, not the source of truth.

One line to say in the room

“Treat prompts, tools, models, and policy as executable behavior: block on critical safety or authorization failures, prove the rest on versioned replays, then canary behind a real rollback switch.”