Agent safety controls

At 3:07 a.m., a support agent notices a duplicate paid charge on a customer’s $840 invoice. It has access to the billing system, the customer record, and the email tool. The model reasons plausibly:

1. The second captured charge is probably accidental.
2. The customer should be refunded.
3. The duplicate record can be deleted.
4. The customer should be told the problem is fixed.

That chain could produce an $840 refund, destroy the evidence needed by finance, and send a confident email before anyone wakes up.

Nothing in that story requires a malicious model. The model can follow its instructions, call the right tools, and still make a costly decision from incomplete information. This is the uncomfortable centre of agent safety: a model can be “working correctly” while the system is doing the wrong thing.

An autonomous agent repeatedly chooses actions from a goal and observations, often by calling tools. A side effect changes something outside the model: sending an email, changing a database row, or charging a card. Engineered safety controls restrict, check, observe, stop, and recover those side effects.

A prompt is not an engineered control. “Be careful with payments” lives inside the same fallible process that decides to make the payment.

Propose-then-commit: gate the irreversible

The most useful pattern separates proposing an action from committing it.

A proposal describes what the agent wants to do. A commit performs the operation against the billing system, database, deployment platform, or customer. Low-risk actions can pass through automatically. High-impact actions pause at a gate for a human or policy service.

The model must not be the final authority over its own permissions. It may say, “This refund is safe,” but the execution boundary must decide using rules and current facts outside the model’s text.

IRREVERSIBLE = {"delete_database", "send_payment", "email_external"}

def execute(action):
    if action in IRREVERSIBLE:
        return f"{action}: PAUSED -> needs approval"
    return f"{action}: auto-executed"

for action in ["read_file", "summarize", "send_payment", "delete_database", "list_dir"]:
    print(" ", execute(action))

  read_file: auto-executed
  summarize: auto-executed
  send_payment: PAUSED -> needs approval
  delete_database: PAUSED -> needs approval
  list_dir: auto-executed

The example is deliberately simple. In production, the gate should inspect a structured action and sit in front of the real tool:

The model may propose a tool call. Only the execution service may cause the side effect.

A useful proposal includes:

• the operation and exact target;
• arguments, such as amount $840;
• reason and supporting evidence;
• identity and authority under which it would run;
• an idempotency key;
• an expiry time.

An idempotency key lets a correctly implemented recipient deduplicate retries. It works only when that recipient durably enforces the key for the same operation, with a defined scope and lifetime. If the provider lacks idempotency, or the key has expired, reconcile its state before retrying and use a compensating workflow if the wrong effect already occurred.

The gate checks facts the model cannot rewrite with a persuasive paragraph:

• Is this agent allowed to use this tool?
• Is the target inside its permitted tenant, account, or environment?
• Is the amount within its limit?
• Is the request still current?
• Has the action already committed?
• Does it require a human, a second service, or a stronger identity?
• Is it reversible?

Approval must bind to the exact proposal. “Approve the refund” is too vague. Approval for invoice inv_4821, amount $840, issued at 03:11, must not silently authorise $8,400 against invoice inv_9270. Before commit, the executor rechecks the policy, target, amount, and approval expiry.

A recheck alone cannot prevent a time-of-check to time-of-use race: facts can change between the recheck and the side effect. Where possible, use an atomic check-and-commit, such as a transaction or conditional write against the approved record version. For external systems, use provider-side idempotency and conditional execution. Otherwise reconcile state and compensate when necessary.

A gate with numbers

Return to the duplicate invoice. Suppose the billing policy has these tiers:

Thresholds are business decisions, not universal safety constants. The durable risk axes are impact, reversibility, uncertainty, and reach: how much can change, how well it can be restored, how ambiguous the evidence is, and how many records or users one action affects.

For the $840 refund, the agent can investigate invoices and prepare a recommendation but cannot commit the refund. The scenario is a duplicate paid charge: the customer was charged twice for the same $840 invoice, and the proposal identifies one captured payment to refund. A reviewer sees the invoice IDs, payment reference, proposed amount, evidence, and policy reason.

After approval, the execution service rechecks that the payment is still captured and refundable, has not already been refunded, still matches the approved proposal, and that approval has not expired. If the duplicate invoice is actually unpaid, the correct operation is usually to void or cancel it, not issue a refund.

This is safer than asking a model for a yes-or-no risk label. The model can explain the proposal; it should not be the only barrier between a paragraph and a payment.

The rest of the toolkit

Propose-then-commit is the central gate, not the whole system. Production autonomy uses layers because each catches a different class of mistake.

Scope first: reduce the blast radius

Least privilege gives an agent only the tools, data, and authority its job requires. The billing support agent might read invoice status and create a refund proposal. It does not need database-admin credentials, unrestricted customer export, or permission to delete rows.

Scope the tool, data, operation, quantity, environment, and identity:

• expose create_refund_draft, not issue_refund;
• restrict records to the assigned tenant or case;
• cap dollars per action and per day;
• keep test agents out of production;
• use a separate, attributable, revocable service identity.

A prompt saying “never access another customer” is not scope. A credential that cannot access another customer is scope.

Gate second: classify outside the model

A policy gate is a trusted service, policy engine, or tested application component that runs after the proposal and before the side effect. Deterministic rules should make deterministic decisions: “refund above $500 requires finance approval” must not depend on whether the model calls the amount “modest.” Use a model to extract information from messy text only when the result is validated against the source record.

Record every denied, paused, and approved proposal, including the decision-maker, policy version, exact arguments, and result. Without that audit trail, “why did it send this?” becomes archaeology.

Monitor third: watch actions, not just answers

A fluent answer can hide a disastrous tool trace. Monitor refunds, deletes, permission changes, retries, unusual tool sequences, approval volume, downstream errors, and invariant violations.

An invariant is a condition that must remain true. “A refund cannot exceed the captured amount” is stronger than asking the model to be sensible: the system can reject it mechanically. Track rare, high-impact events separately from aggregate latency and error rates.

Kill fourth: stop new side effects

A kill switch prevents an agent from starting or committing more work. It may pause one run, version, tenant, or the entire fleet.

It must act at the execution boundary. Pausing orchestration does nothing if workers hold credentials and continue calling the billing API. The switch should block new leases, reject new tool calls, drain queued work, and revoke or expire execution capability.

“Instant” has a boundary: a request already accepted by an external provider may finish. The control should stop the next side effect and expose the remaining uncertainty, not pretend to undo an action already sent.

Roll back last: recover what can be recovered

A checkpoint records the state before a risky step. Rollback restores it when possible. External effects require a compensating action, such as a reversal rather than erasing the original payment.

Prefer soft-delete, draft-then-send, transactions for related writes, idempotent operations, tested backups, and small batches. You cannot unsend an email from every inbox, but you may disable a link, send a correction, reverse a payment, or restore a record. Design and test those paths explicitly.

Choosing the control

The question is not “should this agent be autonomous?” It is “which actions deserve which level of autonomy?”

• Read-only investigation: automatic execution with logging.
• Reversible, low-impact change: automatic execution with limits and idempotency.
• Ambiguous or moderate-impact change: proposal plus human or policy approval.
• Irreversible or high-impact change: approval, narrow scope, and second validation.
• Broad production change: canary, invariant checks, monitoring, and rollback.
• Unrecoverable external action: manual workflow or strong dual control.

A human gate adds latency, queue management, and rubber-stamp risk. Use it when expected harm exceeds review cost, not for every file read. If every action demands a human, the queue becomes the new failure mode.

Failure modes you will see first

Dangerous actions bypass the gate

Symptom: a tool call has no policy decision, or the action appears before approval.

Cause: the rule exists only in the prompt or interface; the endpoint remains directly callable.

Fix: put the gate in the tool service or execution gateway. Test direct, retry, and background-worker paths.

The kill switch does not stop calls

Symptom: the dashboard says “paused” while downstream logs show new refunds or queued calls.

Cause: the switch stopped orchestration, but workers cached permission or retained valid credentials.

Fix: reject calls centrally at execution, revoke short-lived capabilities, drain queues, and account for in-flight requests. Exercise the control before an incident.

Rollback succeeds but harm remains

Symptom: internal state is restored, but an email was delivered, payment settled, or third-party record changed.

Cause: local rollback cannot restore an external or irreversible effect.

Fix: store provider references and build reversal or correction workflows. Name residual harm in the runbook instead of hiding it behind “rollback.”

Canary metrics look healthy

Symptom: normal aggregate error rate, but one customer loses access or one high-value payment is wrong.

Cause: aggregate metrics dilute rare, severe actions.

Fix: monitor action-level invariants and high-severity events separately. A small canary must be broad in observation, not merely small in traffic.

The honest trade-off

Controls reduce autonomy by design. Approval adds a person, a policy service adds latency, a canary slows rollout, and a narrow credential can make a useful-looking task fail.

The alternative has a cost too. If a $10,000 mistaken payment has a one percent chance per month, its rough expected monthly loss is $100 before customer trust and investigation time. A review step costing minutes may be cheap by comparison. Make the arithmetic explicit rather than arguing from instinct.

Controls also create false confidence. An approval screen does not repair an incorrect source record. A checkpoint does not restore an email. A kill switch does not cancel a request already accepted elsewhere. Defense in depth gives you overlapping imperfect barriers; it does not make an agent formally verified.

The goal is bounded autonomy: let the agent move quickly where mistakes are cheap, and make expensive mistakes difficult, visible, stoppable, and recoverable.

In one breath

• Prompt-level safety is a request, not a guarantee. Real side effects need controls outside the model.
• Propose-then-commit lets the agent prepare an action while a trusted gate decides whether it may execute.
• Bind approval to exact targets and arguments, then recheck them before commit.
• Use least-privilege scope, a policy gate, action-level monitoring, a real kill switch, and tested checkpoint or rollback paths.
• Base risk on impact, reversibility, uncertainty, and reach. Do not make harmless reads wait for a human.
• Assume something will go wrong; design how far it gets before you notice.

Quick check

Next

Pair these controls with durable execution for persisted state and recovery, agent security for identity and prompt-injection threats, and agent reliability for safe, idempotent side effects.