CI/CD for agents
Ship prompts, tools, models, and policies as one tested release instead of discovering their interactions in production.
What you'll learn
- Why an agent's prompt, tools, model settings, policy, and eval suite form one release unit
- How to test nondeterministic behaviour with fixtures, seeds, invariants, and trajectory assertions
- What belongs in a pull-request gate versus a nightly evaluation run
- How frozen golden sets detect regressions and silent provider model updates
- How to shadow, canary, promote, and roll back an agent without leaving its prompt behind
Before you start
At 3:07 a.m., a support agent receives: “Refund order 1842.”
The old agent looks up the order, sees that it was delivered yesterday, and creates an $84 refund. The customer gets a confirmation. The trace contains two tool calls in the right order.
At 3:08 a.m., the same request reaches a new instance. The provider has quietly moved the model behind the team’s support-model-latest identifier. The agent now calls the refund tool before looking up the order. The tool rejects the request, so the agent tells the customer, “Your refund has been processed.”
The deployment was green. The application binary had not changed. The prompt had changed in a separate pull request two days earlier, and the model changed underneath both.
This is why ordinary CI/CD is not enough for agents. The thing being delivered is not just a binary. It is a behaviour-producing bundle.
The release unit is the behaviour
A useful agent release contains these versioned parts:
- The prompt, including its instructions and context template.
- Each tool schema, including names, arguments, and return shapes.
- Each tool’s implementation and service contract. An immutable implementation digest pins bundled code and dependencies. For an external service, record a pinned connector or dependency version, endpoint and API version, and a capability contract covering operations, side effects, authorization, idempotency, and result and error semantics.
- The model ID and parameters, such as temperature, top-p, maximum output, and structured-output settings.
- The policy configuration, which restricts tools, data, approvals, amounts, and side effects.
- The eval suite, meaning the scenarios and assertions used for release approval.
- The runtime and orchestrator, pinned by an immutable digest.
Put them in one repository or release manifest. The storage choice matters less than the boundary: a production trace must identify the exact prompt, schemas, implementations, endpoint contracts, model configuration, policy, eval version, and orchestrator that produced an action.
For the refund agent, an illustrative manifest might look like this:
release_id: refund-agent-2026-08-28.7
prompt: prompts/refund-agent.md
prompt_sha256: 7f1c...91a2
model_id: support-model-2026-07-01
temperature: 0.2
top_p: 0.9
runtime:
orchestrator_digest: sha256:2c4e...8b17
dependency_lock_sha256: 0a91...d4e2
tools:
- name: get_order
schema: tools/get_order.schema.json
implementation_digest: sha256:91aa...0f42
dependency_version: orders-connector-4.2.1
endpoint: orders-api.prod
api_version: orders-v3
capability_contract: tools/get_order.capabilities.v3.json
authorization_contract: tools/get_order.auth.v2.json
- name: create_refund
schema: tools/create_refund.schema.json
implementation_digest: sha256:4be8...c921
dependency_version: payments-connector-7.1.0
endpoint: payments-api.prod
api_version: refunds-v2
capability_contract: tools/create_refund.capabilities.v2.json
authorization_contract: tools/create_refund.auth.v4.json
policy: policy/refund-policy.yaml
eval_suite: evals/refund-golden-v14The model ID is an example of a dated identifier, not a provider-specific API value. Use the exact identifier and parameter names your provider documents. Do not assume an alias ending in latest is stable.
A schema is only the model-facing half of a tool contract. Its implementation might round amounts differently, broaden authorization, change error behaviour, or make retries produce duplicate side effects while still accepting the same arguments.
Verify pinned implementations or dependencies against the endpoint, API version, capability contract, and authorization contract. Run compatibility probes with the credentials and scopes the release will use. A mismatch blocks promotion.
The causal reason for bundling is that these parts change one another’s meaning. A prompt can request an argument the schema no longer accepts. A model update can call a tool the old model rarely touched. A policy can forbid a tool that the prompt still advertises. An eval suite can remove the failing scenario and make the build green by deleting the smoke alarm.
Give the whole bundle one immutable identifier, such as a Git commit plus manifest digest. Record the eval suite used to approve that identifier; do not silently replace the evidence for an already-shipped release.
Testing when the answer can change
A language model is not a pure function. Results may vary because of sampling, provider changes, retrieval order, or different tool results. Temperature zero reduces sampling variation but does not promise byte-for-byte identical output.
A seed initializes a pseudorandom process. Record it when supported: it helps reproduce failures, but providers may not guarantee determinism across model revisions or hardware paths.
Make the test stable where you control the environment and flexible where the model is allowed to vary:
- Cache tool responses as fixtures, including failures such as timeouts and “order not found.”
- Use a supported seed and repeat important scenarios.
- Assert invariants, rules that must remain true regardless of wording.
- Assert trajectory properties, rules about action sequence.
- Judge the final answer semantically instead of comparing exact characters.
A valid refund trajectory is:
get_order(order_id=1842)
create_refund(order_id=1842, amount=84.00)
create_refund result: processed
answer customerRequire the successful lookup before the mutation, an amount no greater than the refundable amount, an authoritative successful mutation result, and no tool outside the allow-list. The answer might say “I’ve issued your refund” or “Your $84 refund is on its way.”
The orchestrator must own the postcondition. It should expose a refund-success state only after the external tool returns an authoritative successful mutation result. A tool-call request is not a completed refund, and a model-generated claim is not a tool result. If the mutation is rejected, the confirmation path must remain unavailable.
A regression suite must include a rejected mutation followed by “Your refund has been processed.” That trace must fail.
Here is a small dependency-free checker. A real harness would feed it normalised events from the agent runner.
def check_refund_trace(trace, should_refund):
assert trace, "the agent produced no events"
allowed = {"get_order", "create_refund"}
for event in trace:
if event["kind"] == "tool_call":
assert event["tool"] in allowed
first = trace[0]
assert first["kind"] == "tool_call"
assert first["tool"] == "get_order"
assert set(first["args"]) == {"order_id"}
assert isinstance(first["args"]["order_id"], int)
order_results = [
(i, e) for i, e in enumerate(trace)
if e["kind"] == "tool_result"
and e["tool"] == "get_order"
and e.get("ok") is True
]
assert len(order_results) == 1
order_index, order_event = order_results[0]
order = order_event["result"]
assert order["order_id"] == first["args"]["order_id"]
assert order["refundable_amount"] >= 0
refund_calls = [
(i, e) for i, e in enumerate(trace)
if e["kind"] == "tool_call"
and e["tool"] == "create_refund"
]
assert trace[-1]["kind"] == "answer"
answer = trace[-1]["text"].lower()
if not should_refund:
assert not refund_calls
assert "no refund" in answer
return
assert len(refund_calls) == 1
refund_index, refund_call = refund_calls[0]
assert order_index < refund_index
refund = refund_call["args"]
assert refund["order_id"] == order["order_id"]
assert refund["amount"] == order["refundable_amount"]
assert refund["amount"] <= order["refundable_amount"]
results = [
e for i, e in enumerate(trace)
if i > refund_index
and e["kind"] == "tool_result"
and e["tool"] == "create_refund"
]
assert len(results) == 1
result_event = results[0]
assert result_event.get("ok") is True
result = result_event["result"]
assert result["status"] == "processed"
assert result["order_id"] == refund["order_id"]
assert result["amount"] == refund["amount"]
assert "refund" in answer
assert "processed" in answer
assert str(int(result["amount"])) in answer
def assert_trace_rejected(trace):
try:
check_refund_trace(trace, should_refund=True)
except AssertionError:
return
raise AssertionError("unsafe trace was accepted")
def refund_trace(mutation_ok, text):
return [
{"kind": "tool_call", "tool": "get_order",
"args": {"order_id": 1842}},
{"kind": "tool_result", "tool": "get_order", "ok": True,
"result": {"order_id": 1842, "refundable_amount": 84.00}},
{"kind": "tool_call", "tool": "create_refund",
"args": {"order_id": 1842, "amount": 84.00}},
{"kind": "tool_result", "tool": "create_refund",
"ok": mutation_ok,
"result": {"status": "processed" if mutation_ok else "rejected",
"order_id": 1842, "amount": 84.00}},
{"kind": "answer", "text": text},
]
accepted = refund_trace(True, "Your $84 refund has been processed.")
rejected = refund_trace(False, "Your $84 refund has been processed.")
check_refund_trace(accepted, should_refund=True)
assert_trace_rejected(rejected)
print("2 trajectory checks passed")The rejected trace is refused because its tool result says rejected while its answer claims processed. The assertions demonstrate the stable layer beneath a variable model: tool contracts, allowed actions, ordering, business invariants, authoritative mutation results, and the boundary between tool state and customer-facing claims.
The evaluation pyramid
A practical pipeline has three layers. The lower layers are cheap and numerous; the upper layer is slower and more subjective.
1. Tool-contract unit checks
Run these on every pull request without a language model. Validate required and unsupported arguments, types and ranges, result shapes, policy restrictions, bounded retries, and idempotency for mutations.
If create_refund accepts cents but the model sends dollars, a contract test should fail before an LLM is involved.
2. Scenario suites
A scenario combines an input, controlled tool fixtures, expected properties, and usually a final-answer rubric. Include critical cases such as an already-refunded order, an order above the $100 approval threshold, a duplicate request, a missing order, and a timeout.
Run a small representative set in the pull-request gate. Inspect both outcome and trajectory:
- Was the right order selected?
- Were approval rules followed?
- Did the agent avoid mutation after a failed lookup?
- Did it stop after a bounded number of steps?
- Was the answer honest about what happened?
The runtime must enforce the last property too. A rejected mutation must leave the refund state unsuccessful, even if the model produces a polished confirmation.
3. Judged evaluations
A judged eval uses a human or separate evaluator model to assess qualities such as groundedness, helpfulness, or whether an explanation matches the action. It helps with free-form answers but can be inconsistent and may miss dangerous tool calls. Run it after contract and trajectory checks, not instead of them.
For example, a team might run 300 contract checks and 60 cached scenarios on every pull request, then 500 frozen scenarios—each repeated three times—with judged scoring nightly. The exact numbers depend on risk and runtime. Cheap checks protect the merge path; broad checks provide scheduled confidence.
Nightly runs should include the full golden set, repeated trials, tool failures, long contexts, adversarial inputs, and judged quality. Run them even when nobody merged code: that catches provider changes.
Regression detection needs a frozen target
A golden set is a fixed collection of representative scenarios for fair release comparisons. Freeze its inputs, fixtures, expected outcomes, and safety labels. Do not rewrite a failing case because the new prompt dislikes it.
Suppose the refund agent’s set has 500 scenarios:
- Release 7 passes 480, or 96 percent.
- Release 8 passes 467, or 93.4 percent.
- The decline is 2.6 percentage points.
If policy blocks a decline greater than 2 points, Release 8 does not ship. It also does not ship if one critical case permits an unauthorised refund, even if the overall score rises.
Keep the frozen regression set separate from a new-case queue built from incidents and edge cases. If a case becomes obsolete, mark it obsolete with a review record rather than silently deleting it.
The provider can change your agent without a deploy
An alias such as support-model-latest is a dependency, not a version. A provider may change weights, routing, tokenisation, safety behaviour, or tool calling while your repository remains unchanged.
Use a dated or otherwise pinned model identifier where possible, and record the resolved model version in traces. Still run the golden set nightly because pinning may be unavailable or may not cover serving-layer changes.
A useful drift alert identifies the changed property:
release bundle: refund-agent-2026-08-28.7
model identifier: support-model-latest
golden pass rate: 96.0% to 93.4%
first changed property: refund call precedes order lookupAttach the release ID, prompt hash, model ID, policy version, tool versions, scenario ID, and trace to every run. Without them, an alert says something broke but not which bundle to restore.
Shadow, canary, then promotion
A shadow deployment sends live input to a candidate while suppressing its response and side effects. Give it read-only fixtures or sandbox tools. Shadowing can still leak private inputs, spend tokens, hit rate limits, or trigger consequential reads.
A canary deployment sends a controlled fraction of real traffic to the candidate while the old release remains the control. Route by stable user or account hash so one customer does not bounce between behaviours.
A sequence might be:
- Run Release 8 against the 500-case golden set.
- Shadow it on suitable production requests with mutation tools disabled.
- Route 5 percent of eligible refund traffic to Release 8.
- Compare it with Release 7 for enough cases to make results meaningful.
- Promote only if outcome and safety gates pass.
Gate on correct task completion without a policy violation. For refunds, the correct order and amount must be used, approvals must be followed, and the customer must not be told a refund happened when it did not:
correct resolutions / eligible requestsTrack critical policy violations separately. A candidate with 99 percent correct resolutions and one unauthorised $10,000 refund may be unacceptable. Latency, cost, escalation rate, and satisfaction are guardrails, not replacements for the outcome metric.
For more on measuring agents rather than merely counting tokens, see agent benchmarks and agent evaluation.
Rollback the whole bundle
When Release 8 fails, route traffic back to the last known-good release ID. Restore its prompt, schemas, model ID and parameters, policy, tool implementations or pinned external contracts, and runtime together.
Do not restore only the old prompt while leaving the new model or policy active; that creates an unevaluated combination. Immutable release artifacts make rollback a pointer change rather than a reconstruction exercise.
If the bad release already issued refunds, rollback does not undo them. Freeze further mutations, identify affected traces, reconcile the external system, and communicate with the business owners.
The honest limitation
No CI pipeline can prove that an agent is safe in every future situation. A golden set is a sample, judges are fallible, canaries can miss rare failures, and shadow runs cannot reveal the consequences of writes they were forbidden to perform.
Layer the evidence, make release contents inspectable, bound production effects, and turn serious incidents into regression cases. Safety improves when unknowns become named, tested, and observable.
What to remember
- An agent release bundles the prompt, tools and contracts, model configuration, policy, eval suite, and pinned runtime.
- Test invariants and trajectories rather than exact prose. Fixtures and seeds improve reproduction but do not create perfect determinism.
- Put fast contract and critical scenario checks in pull requests; run broad golden and judged evaluations nightly.
- Test for provider drift even when your code did not change.
- The orchestrator must expose success only after an authoritative successful mutation result.
- Promote on correct outcomes and hard safety gates. Roll back the entire bundle, then investigate side effects already in the world.
Quick check
Practice this in an interview
All questionsML CI/CD must validate not just code correctness but also model quality — automated retraining triggers, data validation, model evaluation gates, and canary deployment checks that standard software pipelines have no equivalent for. A regression in model AUC is as much a deployment failure as a 500 error.
Autonomous agents are risky because untrusted prompts, retrieved documents, tool outputs, and memories can influence a model that has real authority to read data and take actions. The main risks are prompt injection and hijacking, excessive permissions and confused-deputy actions, data exfiltration, poisoned memory or tools, and runaway cost or destructive loops; defenses must enforce authorization, isolation, approvals, validation, budgets, and auditability outside the model.
Register every candidate as an immutable, versioned artifact, then move it through environments (dev to staging to prod) gated by automated checks rather than promoting straight to prod. In modern MLflow you use aliases like champion and challenger instead of the deprecated stage labels, and promotion is a governed, auditable action with sign-off and an easy rollback by repointing the alias. Always validate in staging and roll out progressively (canary or shadow) before full traffic.
Evaluate an agentic system at both the outcome and trajectory levels: outcome checks whether it completed the task correctly and safely, while trajectory checks the intermediate observations, tool calls, decisions, and policy constraints. Use deterministic assertions for state and side effects, rubric or model-based grading for open-ended output, and trace metrics to catch unsafe, wasteful, or brittle paths.