Where would you place input filters, output filters, tool-call checks, and a model such as Llama Guard in an agentic system? How would you manage false positives, latency, adversarial adaptation, and unequal refusal rates across user groups?
Put input checks at ingress, output checks before delivery, and enforce tool-call policy immediately before every side effect. Use a model such as Llama Guard as one risk signal at several trust boundaries, then manage errors with calibrated thresholds, layered controls, monitoring, adversarial testing, and subgroup fairness audits.
How to think about it
I put input filters at the boundary before prompt construction, output filters immediately before content reaches the user or another system, and tool-call checks immediately before every tool executes. I use a model such as Llama Guard at those boundaries as a classifier and risk signal, not as the authority that decides whether an irreversible action is allowed.
Why the placement matters
An agent has several different boundaries, and they need different controls.
An input filter handles what enters the agent: the user message, uploaded files, retrieved documents, and sometimes conversation history. It can reject obvious abuse, remove secrets, enforce size limits, and classify safety-sensitive content before that content is placed in the model’s context.
That last detail matters. A filter applied only to the visible user message misses a malicious instruction hidden in a PDF, web page, email, or database row. In an agent, retrieved content is data, not trusted instructions. The model may not reliably maintain that distinction on its own.
An output filter runs after generation and before the answer is delivered. It can detect a disallowed response, accidental personal data, credentials, or a format that would be dangerous to a downstream system. It is a final check, not a repair shop. If the model has already sent an email or called a payment API, filtering the later explanation is rather beside the point.
A tool-call check is the most important control for actions with side effects. It parses the proposed function name and arguments, validates their types and limits, checks the user’s authorization against the target resource, and applies business policy. It must run in the application or policy service that owns the tool, not only in a system prompt.
I would treat that service as a reference monitor, meaning a component that mediates every operation. The model can propose refund_customer; it must never be able to execute it merely because it produced convincing JSON.
A model such as Llama Guard is useful for semantic classification: for example, deciding whether text appears to request dangerous assistance or contains a category of sensitive content. It is still a probabilistic model. It can miss an attack, misunderstand a benign request, or be manipulated by encoding, translation, and multi-turn decomposition. It should supply evidence to a policy decision, alongside deterministic checks and authorization data.
The important production pattern is therefore:
- Check untrusted content when it enters.
- Check every model response before delivery.
- Check every tool call before execution.
- Check tool results and retrieved content before they are fed back into the next model turn.
- Recheck the final answer, because the agent may have combined several harmless-looking pieces into a harmful result.
A concrete example
Suppose we operate an expense agent for 10,000 employees. It can read receipts, search approved vendors, submit reimbursements, and send email. The product has a 300 millisecond target for an ordinary answer, but submitting a reimbursement is more important than shaving 100 milliseconds from the chat response.
A sensible request might travel like this:
| Boundary | Check | Failure behavior |
|---|---|---|
| User message and upload | Size, malware, secrets, safety category | Reject, redact, or ask for a safer reformulation |
| Model response | Sensitive data and unsafe instructions | Suppress or regenerate with a safe response |
| Tool call | Schema, authorization, amount, destination, confirmation | Deny, require confirmation, or send to review |
| Tool result | Prompt injection, secrets, unexpected content | Treat as untrusted and keep it out of instructions |
For example, the user asks, “Submit my reimbursement for the hotel in Berlin.” The model may identify a receipt and propose a call with an amount of 480 dollars. The tool service then checks that the employee owns the expense, the receipt matches the vendor, the currency is accepted, and the amount is below the employee’s automatic-submission limit.
The policy should not be hidden in the prompt. A small application-side check might look like this:
def tool_decision(call, user):
name = call["name"]
args = call["arguments"]
if name not in {"read_receipt", "submit_reimbursement"}:
return "deny"
if name == "submit_reimbursement":
if args["employee_id"] != user["employee_id"]:
return "deny"
if args["amount_cents"] > 50000:
return "confirm"
return "allow"The 50,000-cent limit is a product policy in this example, not a property of the language model. A real implementation would also validate numeric types, currency, duplicate submissions, receipt ownership, and the authorization service’s answer. A request to change a bank account should probably require stronger authentication and an out-of-band confirmation, regardless of what a safety classifier says.
For latency, I would put cheap deterministic checks first: request size, allowlisted tools, schema validation, and obvious secret patterns. Independent classifiers can run in parallel. A local safety model can handle ordinary traffic, while uncertain or high-impact cases go to a slower review path. The exact latency depends on hardware and model version, so I would measure it in our deployment rather than quote a borrowed benchmark.
The senior-level nuance
The textbook answer is “block unsafe content.” Production needs a decision ladder.
A classifier threshold that catches more harmful requests usually catches more benign ones too. For a low-risk informational request, the response might be allowed, transformed, or sent for review. For a wire transfer, a low-confidence safety result should not be the only reason to block, but a failed authorization check should be decisive.
I would track precision, recall, and abstention by policy category. Precision asks how many blocked items really violated policy. Recall asks how many violations were caught. An abstention is an uncertain case routed to a human or a stronger process. Aggregate “blocked percentage” hides the mistake that matters: a customer support system quietly refusing legitimate requests from one language group, or a finance agent allowing one dangerous action.
Use shadow mode before enforcement. Run the new filter without changing behavior for a representative sample, compare its decisions with reviewed outcomes, and inspect the disagreements. Log category, confidence, policy version, user locale, and outcome, while minimizing or hashing sensitive content. Give users a useful recovery path: “I can’t help with that request” is poor UX when the actual issue was an ambiguous scanned receipt.
Attackers adapt. They will split one harmful request across ten turns, use Unicode lookalikes, translate it, hide instructions in an image, or place them in a tool result. Consequently, do not publish exact thresholds or let users probe a detailed explanation of which detector fired. Canonicalize inputs, test multilingual and multimodal variants, run replay tests against production incidents, and rotate adversarial evaluation sets. Version the policy and classifier separately so a sudden change in refusals can be traced.
Fairness needs its own dashboard. A refusal rate is the share of requests declined; by itself, it is not a fairness metric. Compare matched benign requests across languages, dialects, disability-related wording, and relevant demographic groups. Measure false-positive rates, escalation rates, time to resolution, and the quality of the alternative response. Review intersections, because a problem may appear only for a particular language and age group.
I would not blindly force identical refusal rates. Some populations or workflows genuinely have different risk distributions. The target is consistent treatment of equivalent content and comparable false-positive rates, not cosmetic equality in one aggregate number. If the disparity is real, first improve training and evaluation data, localization, threshold calibration, and human review. Group-specific thresholds can reduce a measured disparity, but they can also encode stereotypes or create legal and governance problems; use them only with explicit policy and legal review.
Finally, measure the system as an agent, not as isolated prompts. A model can pass an input test, pass an output test, and still cause harm through a sequence of individually acceptable tool calls. The side-effect boundary is where the strongest guarantees belong.
What they’ll ask next
“Would you put Llama Guard only on the input?”
No. I would use it where semantic risk appears: incoming text, model output, tool arguments, and selected tool results. I would vary the action by risk and latency. Deterministic authorization remains authoritative for permissions.
“What happens when the safety model is unavailable?”
For a harmless read-only answer, I might use a degraded path with strict limits. For a payment, deletion, external message, or permission change, I fail closed or require explicit human approval. Availability policy should follow side-effect risk.
“How do you know a new filter helped?”
Run it in shadow mode, evaluate a fixed benign and adversarial set, inspect precision and recall, and compare subgroup false-positive and escalation rates before and after rollout. A lower incident count is not enough if legitimate users have simply stopped trying.
The line I would use in the room: “Put classifiers around the trust boundaries, but put authorization and side-effect policy in code at the tool boundary; then measure the errors by risk and by user group rather than pretending one threshold is fair or final.”