Guardrails are runtime checks, not prompt instructions

At 3:07 a.m., your customer-support agent receives a refund request.

The system prompt says: “Never refund more than $500. Never change a customer’s bank details. Ask for approval before unusual actions.”

The model replies politely. It explains that the request is unusual. Then it calls issue_refund with an amount of $4,800.

What happens next?

If the tool accepts the request, you have no guardrail. You have a paragraph the model was supposed to remember.

That distinction matters because teams routinely call prompt instructions “guardrails”. They are not. A prompt instruction is a soft constraint: guidance the model may balance against other goals, context, or mistakes. A runtime check is code that runs outside the model, can reject a proposal, and leaves evidence of what happened.

The first can improve behavior. The second can control an action.

Those are different jobs. Confusing them is how a harmless-looking support bot becomes an incident report.

A prompt is guidance, not a veto

A language model does not execute a system prompt as a rules engine. It generates the next token from the context it has been given. The instruction affects that generation, often substantially, but it does not create a protected boundary around the result.

The model might misunderstand “unusual”. It might treat a manager’s message as permission. It might follow a later instruction in retrieved content. It might produce a valid-looking tool call while incorrectly reasoning that the customer has already been verified.

The model can also face competing goals:

• be helpful,
• complete the workflow,
• follow the latest request,
• avoid refusing legitimate customers,
• obey the system instruction.

A soft constraint is one more influence in that competition. Even if it has high priority in the prompt hierarchy, it remains part of the model’s input. It is not a mechanical veto.

A hard check lives somewhere else. It receives a proposed action and an authoritative context, such as:

• the authenticated user;
• account ownership;
• amount;
• destination;
• approval state.

It returns a decision. The executor obeys that decision without asking the model to explain itself.

The model can argue that the refund is justified. The check does not care. If the policy says the agent may refund at most $500, the executor rejects $4,800.

That is the useful definition of a guardrail: a control that runs at runtime, can fail closed, and produces evidence.

“Fail closed” means that when the check cannot establish that an action is safe, the system refuses, pauses, or routes to a human instead of proceeding. “Produces evidence” means the system records:

• the decision;
• the reason;
• the policy version;
• enough context to investigate it later.

Without those properties, “guardrail” is usually a flattering name for a hope.

Put the check where harm can happen

Three enforcement points

There are three useful places to check an agent:

• before the model sees input;
• before the user sees output;
• before an external action executes.

The third is the critical one.

Imagine the refund agent’s path:

customer request
      |
      v
input check -> model -> output check -> refund tool -> payment system
                                      ^
                                      |
                                action check

An input check examines incoming material before generation. It might:

• reject an oversized request;
• identify an obviously malicious payload;
• remove a known secret.

This can reduce attack surface and save model tokens.

An output check examines text or structured data after generation. It might:

• catch a leaked account number;
• catch an unsupported claim;
• catch a malformed response.

An action check examines the side effect immediately before execution. It verifies what the agent is actually about to do, such as:

• refund money;
• send an email;
• delete a record;
• change a permission;
• publish a message.

The action boundary

The action check is the point at which the system can still prevent the side effect.

Suppose the model writes:

“I cannot refund more than $500, but I have submitted the request for manual review.”

That output sounds safe. But if the model has already called a payment tool, the money may already be moving. An output filter that approves the sentence is describing safety after the important decision has happened.

The same problem appears with streaming. If an agent streams text to a customer while separately invoking tools, checking the final text does not rewind an email that was sent halfway through generation. It does not restore a deleted file. It does not un-send a wire transfer.

The production pattern is straightforward:

model proposes action
      |
      v
parse and normalize
      |
      v
authenticate and authorize
      |
      v
apply deterministic policy
      |
      v
run risk classifier when needed
      |
      v
execute only the approved action
      |
      v
record decision and outcome

The model should propose an action, not possess an unrestricted capability.

For the refund example, the proposal might contain:

• an amount;
• currency;
• customer account;
• reason;
• idempotency key.

The runtime then checks that:

• the customer is authenticated;
• the account belongs to that customer;
• the amount is within the agent’s scope;
• the currency is allowed;
• the request has not already been executed;
• any required approval exists.

The payment service should receive the approved, normalized action from this path. It should not receive a raw blob of model-generated arguments plus a request to “use your judgment”.

This is the same boundary thinking described in agent architecture: the model decides among possibilities, while the surrounding system owns authority.

Deterministic checks and classifier checks fail differently

Not every policy can be expressed as a simple comparison. That does not mean every policy should be handed to another model.

Deterministic rules

A deterministic check has a repeatable rule. Given the same normalized input and policy version, it returns the same answer. Examples include:

• amount is at most $500,
• recipient belongs to the authenticated organization,
• destination domain is on an allowlist,
• requested fields match the approved schema,
• the agent has a write scope for this resource,
• the approval record is present and not expired.

These checks are valuable because their behavior is inspectable. A failed refund limit can produce a precise reason such as amount_exceeds_agent_limit.

But deterministic does not mean complete.

A regular expression that blocks strings containing password will catch one obvious form of secret leakage. It may miss a secret split across tokens, encoded, copied into an image, or described indirectly. A pattern for dangerous shell commands may miss an equivalent command with different whitespace or syntax. A URL allowlist may be bypassed if the system validates the displayed hostname but connects to a different canonical destination.

The failure mode is a predictable gap. You can test it, document it, and often close it with normalization or a better rule. You should not mistake a regex for a semantic understanding of the content.

Classifier thresholds

A classifier check estimates whether something belongs to a category. It may score a message as likely to contain personal data, likely to be abusive, or likely to be a prompt injection. The score is not a verdict from nature. It is a measurement produced by a model at a chosen operating point.

Suppose a classifier scores refund requests from 0 to 1, where higher means “likely fraudulent”. You set the blocking threshold at 0.80.

That threshold creates two errors:

• a legitimate request scoring 0.84 is blocked, a false positive;
• a fraudulent request scoring 0.79 proceeds, a false negative.

Changing the threshold changes the balance. A threshold of 0.60 catches more suspicious requests but inconveniences more legitimate customers. A threshold of 0.95 creates a smoother experience but lets more risk through.

The operating point is a product decision. It depends on the cost of each error, the availability of human review, and the harm caused by a miss. It is not enough to say that the classifier has “high accuracy” on a test set. You need to know what happens at the threshold used in production, on the traffic you actually receive.

For high-consequence actions, a classifier is usually best used as an additional signal or a route to review. A deterministic authorization rule should not be weakened because a classifier thinks the request looks harmless. If the agent lacks permission to change a bank account, a cheerful score of 0.02 should not grant permission.

Fail-open or fail-closed is a product decision

Every runtime check can fail in an operational sense. Its service can time out. A policy store can be unavailable. A model classifier can return an error. A deployment can load the wrong configuration.

You then have a choice.

Fail open means the requested operation proceeds when the check is unavailable. This protects availability, but it turns an outage in the safety system into a possible bypass.

Fail closed means the operation is denied, held, or escalated when the check is unavailable. This protects against bypass, but it can make the product unavailable or frustrating.

There is no universal answer. The correct choice follows from the side effect.

For a low-stakes writing assistant, if an optional toxicity check is unavailable, showing a warning or continuing may be reasonable.

For higher-consequence actions, an unavailable authorization check should not become permission by accident. These actions include:

• a payment refund;
• a permission change;
• a production deployment;
• a deletion.

The choice also need not be global. You can fail closed for money movement and fail open for a non-critical formatting check in the same request. You can allow a draft email to be generated while blocking its send operation. You can queue a refund for review rather than returning a hard error.

Make the behavior explicit. A timeout should create a decision such as policy_unavailable, not silently look like approved.

That distinction helps operators and prevents a dangerous habit: disabling a failing check because the application’s success-rate dashboard looks better without it.

Spend latency and money at the boundary that matters

A check on every call has a cost. It may add network latency, consume classifier tokens, require another service to scale, and create another dependency that can fail.

Suppose an agent handles 100 requests per second and a serial action check adds 80 milliseconds. The check creates about 8 requests’ worth of in-flight waiting at that rate because 100 requests/second × 0.08 seconds = 8 concurrent requests.

Whether that requires more threads, connections, or service capacity depends on the implementation. The waiting does not disappear. Users experience it as extra time before the action completes.

Now suppose an expensive classifier costs $0.001 per call. At 1,000,000 calls per month, that is $1,000 before infrastructure and retries. The arithmetic is simple; the right price depends on the provider and model, so measure your actual bill.

The answer is not to skip checks. It is to spend them where they buy control.

Run cheap structural checks early. Reject an invalid schema before invoking a classifier. Normalize a URL once before checking it in several places. Cache a stable classification when the content and policy context truly match. Run an expensive semantic check only on the path that can create meaningful harm.

Most importantly, do not spend an expensive check on a low-risk response while leaving a high-risk tool call unprotected.

A useful design often has three layers:

1. Cheap input checks reduce obvious abuse and unnecessary model work.
2. Output checks protect what the user receives, especially when leakage or regulated content matters.
3. Deterministic action checks run on every side-effecting tool call, with deeper classification or human approval reserved for risky cases.

The action check may be slower than a prompt instruction. That is acceptable when it protects a $4,800 refund. It is harder to justify when it delays a harmless “Here are three meeting times” response by 300 milliseconds.

The right question is not “Can we remove this check?” It is “Which consequence are we paying to control, and is this the cheapest reliable point at which to control it?”

The agent cost control lesson is relevant here: latency and spend are part of the design, not an embarrassing bill discovered after launch.

A silent block is an invisible bug

A guardrail that blocks without explaining itself teaches the team almost nothing.

The customer sees “Something went wrong.” Support sees a failed request. Engineering sees a retry spike. Nobody knows whether the block came from a refund limit, a classifier false positive, a policy-service timeout, or a malformed model response.

Record every decision

Every decision should produce structured telemetry. At minimum, record:

• a request and trace identifier;
• the action type and normalized fields, with sensitive values redacted or hashed;
• the policy and rule version;
• the decision: allowed, denied, held, or unavailable;
• the reason code;
• check latency;
• whether the decision failed open or closed;
• the downstream outcome;
• whether a human overrode the decision.

Do not log private customer data merely because logging is convenient. Evidence must be useful without becoming a second data-leak problem.

Suppose the refund agent’s denial rate rises from 2% to 18% after a policy deployment. With reason codes, you can see that 15% of requests are failing customer_ownership_mismatch, perhaps because a normalization change strips a leading zero from account identifiers. Without reason codes, the team may loosen the entire guardrail to restore conversion.

This is why agent observability is part of the control, not an afterthought. A block without a reason is a wall. A block with a reason, version, and outcome is a diagnostic instrument.

Observe what gets through

You also need to observe what the check missed. Sample approved actions, review incidents, and compare classifier decisions with later human judgments. A guardrail that reports only denials can look excellent while allowing the dangerous cases through.

The strongest counterargument is still partly right

The best case for prompt-only safety is practical.

Prompts are cheap to change. They add almost no latency. Strong models follow clear instructions surprisingly often. A hard runtime policy can reject legitimate edge cases, create tedious approval queues, and make an agent feel useless. A classifier can be biased, brittle, and expensive. If every harmless answer passes through three services, users will notice.

All true.

Prompt instructions are worth using. They help the model choose safer plans, ask for missing information, avoid unnecessary tool calls, and explain a denial consistently. A model that knows it should never expose a secret is easier to operate than one that must be stopped after generating every unsafe draft.

But those benefits do not make a prompt a control. They make it a good soft layer.

The answer to false positives is not pretending the hard check does not exist. It is choosing where precision matters, adding a review path, tuning the classifier at a measured operating point, and making denials understandable. The answer to latency is not leaving the action boundary open. It is using cheap deterministic checks there and reserving slower checks for higher-risk actions.

Use the prompt to shape behavior. Use runtime checks to enforce authority.

Those roles reinforce each other. They should not be merged in your vocabulary, your threat model, or your incident review.

Capability limits come before runtime checks

A runtime check is not a substitute for limiting what the agent can do.

If the refund agent has credentials that can:

• issue unlimited refunds;
• change bank details;
• export customer records;
• delete accounts;

then your policy layer is carrying far too much responsibility. One missed check, one confused identity mapping, or one implementation bug can turn a small model mistake into a large incident.

Reduce the blast radius

Reduce the blast radius first:

• give the agent only the tools it needs;
• separate read permissions from write permissions;
• cap amounts and rates at the service boundary;
• require a human for irreversible or high-value actions;
• use short-lived, scoped credentials;
• rate-limit repeated attempts;
• make actions idempotent where possible;
• isolate code execution and untrusted content;
• keep a recovery path for mistakes.

A check can stop an action. A capability boundary makes the action impossible or less damaging in the first place. The second is stronger because it does not depend on every policy implementation being perfect.

If the agent is allowed to refund only $500 per transaction and $2,000 per day, the payment service should enforce those limits too. Duplicating the critical limit at the tool or service boundary may feel inelegant. Losing $4,800 because one agent route forgot the check is less elegant.

Agent safety controls covers this broader principle: put authority in boundaries the model cannot rewrite.

What to do on Monday

Pick one agent that can create a real side effect. The refund agent is a good candidate because its failure is easy to price.

Draw the path from model output to external system. Mark the exact line where money, data, messages, or permissions change. If there is no executable check between the model’s proposal and that line, you have found the first control to build.

Then:

1. Replace the free-form tool call with a normalized action schema containing only the fields the tool needs.
2. Add deterministic checks for:
   • identity;
   • authorization;
   • scope;
   • amount;
   • destination;
   • approval;
   • idempotency.
3. Decide the unavailable-check behavior for each action. Write down which paths fail closed and which, if any, may fail open.
4. Add reason codes, policy versions, latency, and outcomes to the decision log. Redact sensitive values.
5. Test both false positives and false negatives. Include:
   • malformed input;
   • encoded variants;
   • prompt injection;
   • duplicate requests;
   • timeouts;
   • policy-service failure.
6. Remove capabilities the agent does not genuinely need. Put important limits in the downstream service as well as the agent runtime.
7. Keep the system prompt. Use it to make the model propose safer actions and explain decisions. Just stop calling it the thing that prevents them.

A prompt can say, “Never refund more than $500.”

A runtime check can make $4,800 impossible.