Agent safety controls

Prompt-level safety — “please be careful, don’t do anything harmful” — is a request, not a guarantee. The moment an agent is autonomous and can take real actions (delete a database, send a payment, deploy code, email a customer), hoping the model behaves is not a safety strategy. You need engineered controls: mechanisms that bound what the agent can do and let you recover when it goes wrong — including the common case where the model is working exactly as trained but still does the wrong thing.

Propose-then-commit: gate the irreversible

The single most important control is to separate proposing an action from executing it for anything high-stakes or irreversible. Low-risk, reversible actions (reading a file, summarizing) run automatically; irreversible ones (sending money, deleting data) are proposed and require approval — from a human, or a policy checker — before they commit:

IRREVERSIBLE = {"delete_database", "send_payment", "email_external"}

def execute(action):
    if action in IRREVERSIBLE:
        return f"{action}: PAUSED -> needs approval"   # propose, don't commit
    return f"{action}: auto-executed"

for a in ["read_file", "summarize", "send_payment", "delete_database", "list_dir"]:
    print(" ", execute(a))

  read_file: auto-executed
  summarize: auto-executed
  send_payment: PAUSED -> needs approval
  delete_database: PAUSED -> needs approval
  list_dir: auto-executed

The reversible actions flow through; the irreversible ones stop at a gate for a human to confirm. This is the same human-in-the-loop idea behind safe deploys — plan, review, then apply — applied to every dangerous tool call.

The rest of the toolkit

Propose-then-commit is one layer; production autonomy stacks several:

• Least privilege / scoping (the recap from prompt injection): give the agent the minimum tools and permissions, so even a wrong action has a bounded blast radius.
• Kill switch. A way to stop an agent instantly — globally or per-agent — when it misbehaves. Non-negotiable for anything autonomous; you must be able to pull the plug without redeploying.
• Canary / staged rollout. Release a new agent or capability to a small slice of traffic first, watch it, then expand — so a regression hits a few requests, not all of them.
• Checkpoints & rollback. Snapshot state before a risky operation so you can undo it, and prefer reversible or transactional operations where possible (soft-delete, draft-then-send). Pairs naturally with durable execution.

In one breath

• Prompt-level safety is a request, not a guarantee; an autonomous agent that takes real actions needs engineered controls to bound it and recover when it errs.
• Propose-then-commit: reversible/low-risk actions auto-execute, but irreversible ones (payments, deletes, deploys) are proposed and require approval before they commit (the demo gates send_payment and delete_database).
• Layer the rest: least-privilege scoping (bounded blast radius), a kill switch (stop instantly), canary/staged rollout (limit a regression’s reach), and checkpoints + rollback (undo; prefer reversible ops).
• This is not the prompt-injection problem — it bounds the agent’s own actions (including honest mistakes), not just adversarial input. You need both.
• The mindset: assume the agent will eventually do something wrong, and engineer so it was gated, contained, stoppable, and undoable.

Quick check

Next

Safety controls pair with durable execution (checkpoints/rollback) and the security lessons — agent security, prompt injection, and credential security — for the full picture of running an autonomous agent safely.